andrey nikolaev latch internals 2012 for ruoug
TRANSCRIPT
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 182
Latch internalsRUOUG Seminar December 6 2012
Andrey NikolaevOracle Database Performance ExpertRDTEX Russia
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 282
WHO AM I
bull AndreyNikolaevrdtexru
bull RDTEX First Line Support Centerbull httpandreynikolaevwordpresscom
bull Latch mutex and beyond
bull Specialize in Oracle performance tuningbull Over 20 years of Oracle related experience as a research
scientist developer DBA performance consultant lecturerhellip
bull Occasionally presented at conferencesbull RUOUG member
RDTEX | Latch internals | RUOUG Seminar
2
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 382
EPISODE OF LATCH CONTENTIONbull Oracle instance hangs due to heavy cache buffers chains latch contention
3
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 482
WHAT WE ALL KNEW ABOUT ORACLE LATCHES
bull Latches spin up to the instance parameter _SPIN_COUNT
that defaults to 2000bull This number can be tuned dynamically
bull Latch waits use increasing exponential backoff
bull Latch waits are very shortbull We should tune application not the _SPIN_COUNT
RDTEX | Latch internals | RUOUG Seminar
4
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 582
CONTEMPORARY LATCHES SINCE 92
RDTEX | Latch internals | RUOUG Seminar
5
I was wondering to find that
bull Exclusive latch spins 10 times morebull The exponential backoff was disappeared more then decade
ago in Oracle 92
bull All except one latch use wait posting now
bull Latch free wait may be infinite
bull Sometimes it is worth to tune the _SPIN_COUNT
bull This parameter is static for most latches
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 682
GREAT SOURCES OF LATCH INFO
ldquoOracle8i Internal Services for Waits Latches Locks and
Memoryrdquo by Steve Adams [1] 1999 ndash Founded new era of Oracle performance tuning
ndash However the Oracle latch technologies were evolved greatly duringthe decade
ldquoSystematic Latch Contention Troubleshootingrdquo approach byTanel Poder [2]
ndash Latchprofxsql script revolutionized the latch tuning
ldquoOracle Core Essential Internals for DBAs and Developersby Jonathan Lewis [3] 2011
RDTEX | Latch internals | RUOUG Seminar
6
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 782
REVIEW OF SERIALIZATION MECHANISMS IN ORACLE
bull Latches are simple low-level serialization mechanisms that coordinatemultiuser access to shared data structures objects and fileshellip
bull A mutual exclusion object (mutex) is a low-level mechanism thatprevents an object in memory from aging out or from being corrupted hellip
bull Internal locks are higher-level more complex mechanisms hellip
Oraclereg Database Concepts 112
bull KGX Mutexes appeared in latest Oracle versions inside Library Cache
RDTEX | Latch internals | RUOUG Seminar
A983139983139983141983155983155
A983139983153983157983145983155983145983156983145983151983150
983123983152983145983150983148983151983139983147 983138983137983155983141983140
983124983145983149983141983155983139983137983148983141
983116983145983142983141983139983161983139983148983141
983116983151983139983147983155
983123983141983158983141983154983137983148 983117983151983140983141983155
FIF983119
983118983151
gt 983117983145983148983148983145983155983141983139983151983150983140983155
D983161983150983137983149983145983139
983116983137983156983139983144983141983155
983124983161983152983141983155 983137983150983140 983117983151983140983141983155
983123I983122983119 (983155983152983145983150) + FIF983119
983129983141983155
983117983145983139983154983151983155983141983139983151983150983140983155
983123983156983137983156983145983139
983117983157983156983141983160983141983155
983119983152983141983154983137983156983145983151983150983155
983123I983122983119
983129983141983155
983123983157983138983117983145983139983154983151983155983141983139983151983150983140983155
D983161983150983137983149983145983139 7
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882
HOW SPINLOCK WORKS
RDTEX | Latch internals | RUOUG Seminar
8
CPU3
Free FreeHolding by CPU 1
Spin polls the spinlock location
Get
Miss Sleep Spin count ( )∆
Holding time (S )
CPU2 Spin Holding the spinlock
Holding by CPU 2
CPU1 Holding the spinlock
The spinlock location
Sleep releases the CPU
bull Oracle latches
ndash Use atomic hardware instruction for Immediate get
ndash If missed process spins by polling latch location during Spin get
ndash Number of spin cycles is bounded by spin count
ndash In spin get not succeed the process acquiring the latch sleeps
hellip
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982
CLASSIC SPINLOCKS
bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip
bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]
bull Many sophisticated spinlock realizations were proposed and evaluated
(TS TTS Delay MCS Anderson)
bull Two general types
ndash System spinlocks Kernel OS threads cannot wait Major metrics
atomic operations frequency Shared bus utilization
ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times
RDTEX | Latch internals | RUOUG Seminar
9
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082
THE NEED FOR SPIN
RDTEX | Latch internals | RUOUG Seminar
10
Time to complete CBC latch contention testcase vs number of threads
No spinning _spin_count=0
_spin_count=2000
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182
SPINLOCK REALIZATIONS
RDTEX | Latch internals | RUOUG Seminar
Spinlock Pseudocode Problems
TS
pre-112 mutex
while(Test_and_Set(lock)) Bus saturation
TTS
Oracle latch
while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo
ldquothundering herdsrdquo)
Delay
Latch and 112
Mutex
Adjustable delay Slower under contention
Anderson MCSetc
Queues Widely used in JavaLinux kernel hellip
CPU and memoryoverhead preemptionissues
11
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282
THE TOOLS
RDTEX | Latch internals | RUOUG Seminar12
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382
DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK
DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle
providermodulefunctionname
pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482
ORADEBUG INTERNAL ORACLE DEBUGGER
RDTEX | Latch internals | RUOUG Seminar
bull oradebug call Allows us invoke any internal Oracle function manually
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1
bull oradebug peek Examines contents of any memory location
bull oradebug poke Modifies memory
bull oradebug watch Watch a region of memory
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582
LATCH CONTENTION TESTCASES
RDTEX | Latch internals | RUOUG Seminar
15
create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention
end loop
bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0
bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies
exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip
exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip
bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 282
WHO AM I
bull AndreyNikolaevrdtexru
bull RDTEX First Line Support Centerbull httpandreynikolaevwordpresscom
bull Latch mutex and beyond
bull Specialize in Oracle performance tuningbull Over 20 years of Oracle related experience as a research
scientist developer DBA performance consultant lecturerhellip
bull Occasionally presented at conferencesbull RUOUG member
RDTEX | Latch internals | RUOUG Seminar
2
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 382
EPISODE OF LATCH CONTENTIONbull Oracle instance hangs due to heavy cache buffers chains latch contention
3
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 482
WHAT WE ALL KNEW ABOUT ORACLE LATCHES
bull Latches spin up to the instance parameter _SPIN_COUNT
that defaults to 2000bull This number can be tuned dynamically
bull Latch waits use increasing exponential backoff
bull Latch waits are very shortbull We should tune application not the _SPIN_COUNT
RDTEX | Latch internals | RUOUG Seminar
4
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 582
CONTEMPORARY LATCHES SINCE 92
RDTEX | Latch internals | RUOUG Seminar
5
I was wondering to find that
bull Exclusive latch spins 10 times morebull The exponential backoff was disappeared more then decade
ago in Oracle 92
bull All except one latch use wait posting now
bull Latch free wait may be infinite
bull Sometimes it is worth to tune the _SPIN_COUNT
bull This parameter is static for most latches
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 682
GREAT SOURCES OF LATCH INFO
ldquoOracle8i Internal Services for Waits Latches Locks and
Memoryrdquo by Steve Adams [1] 1999 ndash Founded new era of Oracle performance tuning
ndash However the Oracle latch technologies were evolved greatly duringthe decade
ldquoSystematic Latch Contention Troubleshootingrdquo approach byTanel Poder [2]
ndash Latchprofxsql script revolutionized the latch tuning
ldquoOracle Core Essential Internals for DBAs and Developersby Jonathan Lewis [3] 2011
RDTEX | Latch internals | RUOUG Seminar
6
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 782
REVIEW OF SERIALIZATION MECHANISMS IN ORACLE
bull Latches are simple low-level serialization mechanisms that coordinatemultiuser access to shared data structures objects and fileshellip
bull A mutual exclusion object (mutex) is a low-level mechanism thatprevents an object in memory from aging out or from being corrupted hellip
bull Internal locks are higher-level more complex mechanisms hellip
Oraclereg Database Concepts 112
bull KGX Mutexes appeared in latest Oracle versions inside Library Cache
RDTEX | Latch internals | RUOUG Seminar
A983139983139983141983155983155
A983139983153983157983145983155983145983156983145983151983150
983123983152983145983150983148983151983139983147 983138983137983155983141983140
983124983145983149983141983155983139983137983148983141
983116983145983142983141983139983161983139983148983141
983116983151983139983147983155
983123983141983158983141983154983137983148 983117983151983140983141983155
FIF983119
983118983151
gt 983117983145983148983148983145983155983141983139983151983150983140983155
D983161983150983137983149983145983139
983116983137983156983139983144983141983155
983124983161983152983141983155 983137983150983140 983117983151983140983141983155
983123I983122983119 (983155983152983145983150) + FIF983119
983129983141983155
983117983145983139983154983151983155983141983139983151983150983140983155
983123983156983137983156983145983139
983117983157983156983141983160983141983155
983119983152983141983154983137983156983145983151983150983155
983123I983122983119
983129983141983155
983123983157983138983117983145983139983154983151983155983141983139983151983150983140983155
D983161983150983137983149983145983139 7
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882
HOW SPINLOCK WORKS
RDTEX | Latch internals | RUOUG Seminar
8
CPU3
Free FreeHolding by CPU 1
Spin polls the spinlock location
Get
Miss Sleep Spin count ( )∆
Holding time (S )
CPU2 Spin Holding the spinlock
Holding by CPU 2
CPU1 Holding the spinlock
The spinlock location
Sleep releases the CPU
bull Oracle latches
ndash Use atomic hardware instruction for Immediate get
ndash If missed process spins by polling latch location during Spin get
ndash Number of spin cycles is bounded by spin count
ndash In spin get not succeed the process acquiring the latch sleeps
hellip
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982
CLASSIC SPINLOCKS
bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip
bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]
bull Many sophisticated spinlock realizations were proposed and evaluated
(TS TTS Delay MCS Anderson)
bull Two general types
ndash System spinlocks Kernel OS threads cannot wait Major metrics
atomic operations frequency Shared bus utilization
ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times
RDTEX | Latch internals | RUOUG Seminar
9
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082
THE NEED FOR SPIN
RDTEX | Latch internals | RUOUG Seminar
10
Time to complete CBC latch contention testcase vs number of threads
No spinning _spin_count=0
_spin_count=2000
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182
SPINLOCK REALIZATIONS
RDTEX | Latch internals | RUOUG Seminar
Spinlock Pseudocode Problems
TS
pre-112 mutex
while(Test_and_Set(lock)) Bus saturation
TTS
Oracle latch
while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo
ldquothundering herdsrdquo)
Delay
Latch and 112
Mutex
Adjustable delay Slower under contention
Anderson MCSetc
Queues Widely used in JavaLinux kernel hellip
CPU and memoryoverhead preemptionissues
11
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282
THE TOOLS
RDTEX | Latch internals | RUOUG Seminar12
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382
DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK
DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle
providermodulefunctionname
pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482
ORADEBUG INTERNAL ORACLE DEBUGGER
RDTEX | Latch internals | RUOUG Seminar
bull oradebug call Allows us invoke any internal Oracle function manually
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1
bull oradebug peek Examines contents of any memory location
bull oradebug poke Modifies memory
bull oradebug watch Watch a region of memory
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582
LATCH CONTENTION TESTCASES
RDTEX | Latch internals | RUOUG Seminar
15
create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention
end loop
bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0
bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies
exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip
exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip
bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 382
EPISODE OF LATCH CONTENTIONbull Oracle instance hangs due to heavy cache buffers chains latch contention
3
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 482
WHAT WE ALL KNEW ABOUT ORACLE LATCHES
bull Latches spin up to the instance parameter _SPIN_COUNT
that defaults to 2000bull This number can be tuned dynamically
bull Latch waits use increasing exponential backoff
bull Latch waits are very shortbull We should tune application not the _SPIN_COUNT
RDTEX | Latch internals | RUOUG Seminar
4
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 582
CONTEMPORARY LATCHES SINCE 92
RDTEX | Latch internals | RUOUG Seminar
5
I was wondering to find that
bull Exclusive latch spins 10 times morebull The exponential backoff was disappeared more then decade
ago in Oracle 92
bull All except one latch use wait posting now
bull Latch free wait may be infinite
bull Sometimes it is worth to tune the _SPIN_COUNT
bull This parameter is static for most latches
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 682
GREAT SOURCES OF LATCH INFO
ldquoOracle8i Internal Services for Waits Latches Locks and
Memoryrdquo by Steve Adams [1] 1999 ndash Founded new era of Oracle performance tuning
ndash However the Oracle latch technologies were evolved greatly duringthe decade
ldquoSystematic Latch Contention Troubleshootingrdquo approach byTanel Poder [2]
ndash Latchprofxsql script revolutionized the latch tuning
ldquoOracle Core Essential Internals for DBAs and Developersby Jonathan Lewis [3] 2011
RDTEX | Latch internals | RUOUG Seminar
6
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 782
REVIEW OF SERIALIZATION MECHANISMS IN ORACLE
bull Latches are simple low-level serialization mechanisms that coordinatemultiuser access to shared data structures objects and fileshellip
bull A mutual exclusion object (mutex) is a low-level mechanism thatprevents an object in memory from aging out or from being corrupted hellip
bull Internal locks are higher-level more complex mechanisms hellip
Oraclereg Database Concepts 112
bull KGX Mutexes appeared in latest Oracle versions inside Library Cache
RDTEX | Latch internals | RUOUG Seminar
A983139983139983141983155983155
A983139983153983157983145983155983145983156983145983151983150
983123983152983145983150983148983151983139983147 983138983137983155983141983140
983124983145983149983141983155983139983137983148983141
983116983145983142983141983139983161983139983148983141
983116983151983139983147983155
983123983141983158983141983154983137983148 983117983151983140983141983155
FIF983119
983118983151
gt 983117983145983148983148983145983155983141983139983151983150983140983155
D983161983150983137983149983145983139
983116983137983156983139983144983141983155
983124983161983152983141983155 983137983150983140 983117983151983140983141983155
983123I983122983119 (983155983152983145983150) + FIF983119
983129983141983155
983117983145983139983154983151983155983141983139983151983150983140983155
983123983156983137983156983145983139
983117983157983156983141983160983141983155
983119983152983141983154983137983156983145983151983150983155
983123I983122983119
983129983141983155
983123983157983138983117983145983139983154983151983155983141983139983151983150983140983155
D983161983150983137983149983145983139 7
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882
HOW SPINLOCK WORKS
RDTEX | Latch internals | RUOUG Seminar
8
CPU3
Free FreeHolding by CPU 1
Spin polls the spinlock location
Get
Miss Sleep Spin count ( )∆
Holding time (S )
CPU2 Spin Holding the spinlock
Holding by CPU 2
CPU1 Holding the spinlock
The spinlock location
Sleep releases the CPU
bull Oracle latches
ndash Use atomic hardware instruction for Immediate get
ndash If missed process spins by polling latch location during Spin get
ndash Number of spin cycles is bounded by spin count
ndash In spin get not succeed the process acquiring the latch sleeps
hellip
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982
CLASSIC SPINLOCKS
bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip
bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]
bull Many sophisticated spinlock realizations were proposed and evaluated
(TS TTS Delay MCS Anderson)
bull Two general types
ndash System spinlocks Kernel OS threads cannot wait Major metrics
atomic operations frequency Shared bus utilization
ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times
RDTEX | Latch internals | RUOUG Seminar
9
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082
THE NEED FOR SPIN
RDTEX | Latch internals | RUOUG Seminar
10
Time to complete CBC latch contention testcase vs number of threads
No spinning _spin_count=0
_spin_count=2000
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182
SPINLOCK REALIZATIONS
RDTEX | Latch internals | RUOUG Seminar
Spinlock Pseudocode Problems
TS
pre-112 mutex
while(Test_and_Set(lock)) Bus saturation
TTS
Oracle latch
while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo
ldquothundering herdsrdquo)
Delay
Latch and 112
Mutex
Adjustable delay Slower under contention
Anderson MCSetc
Queues Widely used in JavaLinux kernel hellip
CPU and memoryoverhead preemptionissues
11
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282
THE TOOLS
RDTEX | Latch internals | RUOUG Seminar12
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382
DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK
DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle
providermodulefunctionname
pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482
ORADEBUG INTERNAL ORACLE DEBUGGER
RDTEX | Latch internals | RUOUG Seminar
bull oradebug call Allows us invoke any internal Oracle function manually
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1
bull oradebug peek Examines contents of any memory location
bull oradebug poke Modifies memory
bull oradebug watch Watch a region of memory
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582
LATCH CONTENTION TESTCASES
RDTEX | Latch internals | RUOUG Seminar
15
create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention
end loop
bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0
bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies
exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip
exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip
bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 482
WHAT WE ALL KNEW ABOUT ORACLE LATCHES
bull Latches spin up to the instance parameter _SPIN_COUNT
that defaults to 2000bull This number can be tuned dynamically
bull Latch waits use increasing exponential backoff
bull Latch waits are very shortbull We should tune application not the _SPIN_COUNT
RDTEX | Latch internals | RUOUG Seminar
4
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 582
CONTEMPORARY LATCHES SINCE 92
RDTEX | Latch internals | RUOUG Seminar
5
I was wondering to find that
bull Exclusive latch spins 10 times morebull The exponential backoff was disappeared more then decade
ago in Oracle 92
bull All except one latch use wait posting now
bull Latch free wait may be infinite
bull Sometimes it is worth to tune the _SPIN_COUNT
bull This parameter is static for most latches
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 682
GREAT SOURCES OF LATCH INFO
ldquoOracle8i Internal Services for Waits Latches Locks and
Memoryrdquo by Steve Adams [1] 1999 ndash Founded new era of Oracle performance tuning
ndash However the Oracle latch technologies were evolved greatly duringthe decade
ldquoSystematic Latch Contention Troubleshootingrdquo approach byTanel Poder [2]
ndash Latchprofxsql script revolutionized the latch tuning
ldquoOracle Core Essential Internals for DBAs and Developersby Jonathan Lewis [3] 2011
RDTEX | Latch internals | RUOUG Seminar
6
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 782
REVIEW OF SERIALIZATION MECHANISMS IN ORACLE
bull Latches are simple low-level serialization mechanisms that coordinatemultiuser access to shared data structures objects and fileshellip
bull A mutual exclusion object (mutex) is a low-level mechanism thatprevents an object in memory from aging out or from being corrupted hellip
bull Internal locks are higher-level more complex mechanisms hellip
Oraclereg Database Concepts 112
bull KGX Mutexes appeared in latest Oracle versions inside Library Cache
RDTEX | Latch internals | RUOUG Seminar
A983139983139983141983155983155
A983139983153983157983145983155983145983156983145983151983150
983123983152983145983150983148983151983139983147 983138983137983155983141983140
983124983145983149983141983155983139983137983148983141
983116983145983142983141983139983161983139983148983141
983116983151983139983147983155
983123983141983158983141983154983137983148 983117983151983140983141983155
FIF983119
983118983151
gt 983117983145983148983148983145983155983141983139983151983150983140983155
D983161983150983137983149983145983139
983116983137983156983139983144983141983155
983124983161983152983141983155 983137983150983140 983117983151983140983141983155
983123I983122983119 (983155983152983145983150) + FIF983119
983129983141983155
983117983145983139983154983151983155983141983139983151983150983140983155
983123983156983137983156983145983139
983117983157983156983141983160983141983155
983119983152983141983154983137983156983145983151983150983155
983123I983122983119
983129983141983155
983123983157983138983117983145983139983154983151983155983141983139983151983150983140983155
D983161983150983137983149983145983139 7
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882
HOW SPINLOCK WORKS
RDTEX | Latch internals | RUOUG Seminar
8
CPU3
Free FreeHolding by CPU 1
Spin polls the spinlock location
Get
Miss Sleep Spin count ( )∆
Holding time (S )
CPU2 Spin Holding the spinlock
Holding by CPU 2
CPU1 Holding the spinlock
The spinlock location
Sleep releases the CPU
bull Oracle latches
ndash Use atomic hardware instruction for Immediate get
ndash If missed process spins by polling latch location during Spin get
ndash Number of spin cycles is bounded by spin count
ndash In spin get not succeed the process acquiring the latch sleeps
hellip
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982
CLASSIC SPINLOCKS
bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip
bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]
bull Many sophisticated spinlock realizations were proposed and evaluated
(TS TTS Delay MCS Anderson)
bull Two general types
ndash System spinlocks Kernel OS threads cannot wait Major metrics
atomic operations frequency Shared bus utilization
ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times
RDTEX | Latch internals | RUOUG Seminar
9
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082
THE NEED FOR SPIN
RDTEX | Latch internals | RUOUG Seminar
10
Time to complete CBC latch contention testcase vs number of threads
No spinning _spin_count=0
_spin_count=2000
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182
SPINLOCK REALIZATIONS
RDTEX | Latch internals | RUOUG Seminar
Spinlock Pseudocode Problems
TS
pre-112 mutex
while(Test_and_Set(lock)) Bus saturation
TTS
Oracle latch
while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo
ldquothundering herdsrdquo)
Delay
Latch and 112
Mutex
Adjustable delay Slower under contention
Anderson MCSetc
Queues Widely used in JavaLinux kernel hellip
CPU and memoryoverhead preemptionissues
11
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282
THE TOOLS
RDTEX | Latch internals | RUOUG Seminar12
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382
DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK
DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle
providermodulefunctionname
pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482
ORADEBUG INTERNAL ORACLE DEBUGGER
RDTEX | Latch internals | RUOUG Seminar
bull oradebug call Allows us invoke any internal Oracle function manually
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1
bull oradebug peek Examines contents of any memory location
bull oradebug poke Modifies memory
bull oradebug watch Watch a region of memory
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582
LATCH CONTENTION TESTCASES
RDTEX | Latch internals | RUOUG Seminar
15
create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention
end loop
bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0
bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies
exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip
exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip
bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 582
CONTEMPORARY LATCHES SINCE 92
RDTEX | Latch internals | RUOUG Seminar
5
I was wondering to find that
bull Exclusive latch spins 10 times morebull The exponential backoff was disappeared more then decade
ago in Oracle 92
bull All except one latch use wait posting now
bull Latch free wait may be infinite
bull Sometimes it is worth to tune the _SPIN_COUNT
bull This parameter is static for most latches
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 682
GREAT SOURCES OF LATCH INFO
ldquoOracle8i Internal Services for Waits Latches Locks and
Memoryrdquo by Steve Adams [1] 1999 ndash Founded new era of Oracle performance tuning
ndash However the Oracle latch technologies were evolved greatly duringthe decade
ldquoSystematic Latch Contention Troubleshootingrdquo approach byTanel Poder [2]
ndash Latchprofxsql script revolutionized the latch tuning
ldquoOracle Core Essential Internals for DBAs and Developersby Jonathan Lewis [3] 2011
RDTEX | Latch internals | RUOUG Seminar
6
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 782
REVIEW OF SERIALIZATION MECHANISMS IN ORACLE
bull Latches are simple low-level serialization mechanisms that coordinatemultiuser access to shared data structures objects and fileshellip
bull A mutual exclusion object (mutex) is a low-level mechanism thatprevents an object in memory from aging out or from being corrupted hellip
bull Internal locks are higher-level more complex mechanisms hellip
Oraclereg Database Concepts 112
bull KGX Mutexes appeared in latest Oracle versions inside Library Cache
RDTEX | Latch internals | RUOUG Seminar
A983139983139983141983155983155
A983139983153983157983145983155983145983156983145983151983150
983123983152983145983150983148983151983139983147 983138983137983155983141983140
983124983145983149983141983155983139983137983148983141
983116983145983142983141983139983161983139983148983141
983116983151983139983147983155
983123983141983158983141983154983137983148 983117983151983140983141983155
FIF983119
983118983151
gt 983117983145983148983148983145983155983141983139983151983150983140983155
D983161983150983137983149983145983139
983116983137983156983139983144983141983155
983124983161983152983141983155 983137983150983140 983117983151983140983141983155
983123I983122983119 (983155983152983145983150) + FIF983119
983129983141983155
983117983145983139983154983151983155983141983139983151983150983140983155
983123983156983137983156983145983139
983117983157983156983141983160983141983155
983119983152983141983154983137983156983145983151983150983155
983123I983122983119
983129983141983155
983123983157983138983117983145983139983154983151983155983141983139983151983150983140983155
D983161983150983137983149983145983139 7
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882
HOW SPINLOCK WORKS
RDTEX | Latch internals | RUOUG Seminar
8
CPU3
Free FreeHolding by CPU 1
Spin polls the spinlock location
Get
Miss Sleep Spin count ( )∆
Holding time (S )
CPU2 Spin Holding the spinlock
Holding by CPU 2
CPU1 Holding the spinlock
The spinlock location
Sleep releases the CPU
bull Oracle latches
ndash Use atomic hardware instruction for Immediate get
ndash If missed process spins by polling latch location during Spin get
ndash Number of spin cycles is bounded by spin count
ndash In spin get not succeed the process acquiring the latch sleeps
hellip
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982
CLASSIC SPINLOCKS
bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip
bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]
bull Many sophisticated spinlock realizations were proposed and evaluated
(TS TTS Delay MCS Anderson)
bull Two general types
ndash System spinlocks Kernel OS threads cannot wait Major metrics
atomic operations frequency Shared bus utilization
ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times
RDTEX | Latch internals | RUOUG Seminar
9
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082
THE NEED FOR SPIN
RDTEX | Latch internals | RUOUG Seminar
10
Time to complete CBC latch contention testcase vs number of threads
No spinning _spin_count=0
_spin_count=2000
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182
SPINLOCK REALIZATIONS
RDTEX | Latch internals | RUOUG Seminar
Spinlock Pseudocode Problems
TS
pre-112 mutex
while(Test_and_Set(lock)) Bus saturation
TTS
Oracle latch
while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo
ldquothundering herdsrdquo)
Delay
Latch and 112
Mutex
Adjustable delay Slower under contention
Anderson MCSetc
Queues Widely used in JavaLinux kernel hellip
CPU and memoryoverhead preemptionissues
11
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282
THE TOOLS
RDTEX | Latch internals | RUOUG Seminar12
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382
DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK
DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle
providermodulefunctionname
pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482
ORADEBUG INTERNAL ORACLE DEBUGGER
RDTEX | Latch internals | RUOUG Seminar
bull oradebug call Allows us invoke any internal Oracle function manually
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1
bull oradebug peek Examines contents of any memory location
bull oradebug poke Modifies memory
bull oradebug watch Watch a region of memory
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582
LATCH CONTENTION TESTCASES
RDTEX | Latch internals | RUOUG Seminar
15
create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention
end loop
bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0
bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies
exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip
exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip
bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 682
GREAT SOURCES OF LATCH INFO
ldquoOracle8i Internal Services for Waits Latches Locks and
Memoryrdquo by Steve Adams [1] 1999 ndash Founded new era of Oracle performance tuning
ndash However the Oracle latch technologies were evolved greatly duringthe decade
ldquoSystematic Latch Contention Troubleshootingrdquo approach byTanel Poder [2]
ndash Latchprofxsql script revolutionized the latch tuning
ldquoOracle Core Essential Internals for DBAs and Developersby Jonathan Lewis [3] 2011
RDTEX | Latch internals | RUOUG Seminar
6
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 782
REVIEW OF SERIALIZATION MECHANISMS IN ORACLE
bull Latches are simple low-level serialization mechanisms that coordinatemultiuser access to shared data structures objects and fileshellip
bull A mutual exclusion object (mutex) is a low-level mechanism thatprevents an object in memory from aging out or from being corrupted hellip
bull Internal locks are higher-level more complex mechanisms hellip
Oraclereg Database Concepts 112
bull KGX Mutexes appeared in latest Oracle versions inside Library Cache
RDTEX | Latch internals | RUOUG Seminar
A983139983139983141983155983155
A983139983153983157983145983155983145983156983145983151983150
983123983152983145983150983148983151983139983147 983138983137983155983141983140
983124983145983149983141983155983139983137983148983141
983116983145983142983141983139983161983139983148983141
983116983151983139983147983155
983123983141983158983141983154983137983148 983117983151983140983141983155
FIF983119
983118983151
gt 983117983145983148983148983145983155983141983139983151983150983140983155
D983161983150983137983149983145983139
983116983137983156983139983144983141983155
983124983161983152983141983155 983137983150983140 983117983151983140983141983155
983123I983122983119 (983155983152983145983150) + FIF983119
983129983141983155
983117983145983139983154983151983155983141983139983151983150983140983155
983123983156983137983156983145983139
983117983157983156983141983160983141983155
983119983152983141983154983137983156983145983151983150983155
983123I983122983119
983129983141983155
983123983157983138983117983145983139983154983151983155983141983139983151983150983140983155
D983161983150983137983149983145983139 7
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882
HOW SPINLOCK WORKS
RDTEX | Latch internals | RUOUG Seminar
8
CPU3
Free FreeHolding by CPU 1
Spin polls the spinlock location
Get
Miss Sleep Spin count ( )∆
Holding time (S )
CPU2 Spin Holding the spinlock
Holding by CPU 2
CPU1 Holding the spinlock
The spinlock location
Sleep releases the CPU
bull Oracle latches
ndash Use atomic hardware instruction for Immediate get
ndash If missed process spins by polling latch location during Spin get
ndash Number of spin cycles is bounded by spin count
ndash In spin get not succeed the process acquiring the latch sleeps
hellip
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982
CLASSIC SPINLOCKS
bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip
bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]
bull Many sophisticated spinlock realizations were proposed and evaluated
(TS TTS Delay MCS Anderson)
bull Two general types
ndash System spinlocks Kernel OS threads cannot wait Major metrics
atomic operations frequency Shared bus utilization
ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times
RDTEX | Latch internals | RUOUG Seminar
9
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082
THE NEED FOR SPIN
RDTEX | Latch internals | RUOUG Seminar
10
Time to complete CBC latch contention testcase vs number of threads
No spinning _spin_count=0
_spin_count=2000
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182
SPINLOCK REALIZATIONS
RDTEX | Latch internals | RUOUG Seminar
Spinlock Pseudocode Problems
TS
pre-112 mutex
while(Test_and_Set(lock)) Bus saturation
TTS
Oracle latch
while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo
ldquothundering herdsrdquo)
Delay
Latch and 112
Mutex
Adjustable delay Slower under contention
Anderson MCSetc
Queues Widely used in JavaLinux kernel hellip
CPU and memoryoverhead preemptionissues
11
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282
THE TOOLS
RDTEX | Latch internals | RUOUG Seminar12
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382
DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK
DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle
providermodulefunctionname
pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482
ORADEBUG INTERNAL ORACLE DEBUGGER
RDTEX | Latch internals | RUOUG Seminar
bull oradebug call Allows us invoke any internal Oracle function manually
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1
bull oradebug peek Examines contents of any memory location
bull oradebug poke Modifies memory
bull oradebug watch Watch a region of memory
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582
LATCH CONTENTION TESTCASES
RDTEX | Latch internals | RUOUG Seminar
15
create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention
end loop
bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0
bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies
exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip
exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip
bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 782
REVIEW OF SERIALIZATION MECHANISMS IN ORACLE
bull Latches are simple low-level serialization mechanisms that coordinatemultiuser access to shared data structures objects and fileshellip
bull A mutual exclusion object (mutex) is a low-level mechanism thatprevents an object in memory from aging out or from being corrupted hellip
bull Internal locks are higher-level more complex mechanisms hellip
Oraclereg Database Concepts 112
bull KGX Mutexes appeared in latest Oracle versions inside Library Cache
RDTEX | Latch internals | RUOUG Seminar
A983139983139983141983155983155
A983139983153983157983145983155983145983156983145983151983150
983123983152983145983150983148983151983139983147 983138983137983155983141983140
983124983145983149983141983155983139983137983148983141
983116983145983142983141983139983161983139983148983141
983116983151983139983147983155
983123983141983158983141983154983137983148 983117983151983140983141983155
FIF983119
983118983151
gt 983117983145983148983148983145983155983141983139983151983150983140983155
D983161983150983137983149983145983139
983116983137983156983139983144983141983155
983124983161983152983141983155 983137983150983140 983117983151983140983141983155
983123I983122983119 (983155983152983145983150) + FIF983119
983129983141983155
983117983145983139983154983151983155983141983139983151983150983140983155
983123983156983137983156983145983139
983117983157983156983141983160983141983155
983119983152983141983154983137983156983145983151983150983155
983123I983122983119
983129983141983155
983123983157983138983117983145983139983154983151983155983141983139983151983150983140983155
D983161983150983137983149983145983139 7
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882
HOW SPINLOCK WORKS
RDTEX | Latch internals | RUOUG Seminar
8
CPU3
Free FreeHolding by CPU 1
Spin polls the spinlock location
Get
Miss Sleep Spin count ( )∆
Holding time (S )
CPU2 Spin Holding the spinlock
Holding by CPU 2
CPU1 Holding the spinlock
The spinlock location
Sleep releases the CPU
bull Oracle latches
ndash Use atomic hardware instruction for Immediate get
ndash If missed process spins by polling latch location during Spin get
ndash Number of spin cycles is bounded by spin count
ndash In spin get not succeed the process acquiring the latch sleeps
hellip
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982
CLASSIC SPINLOCKS
bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip
bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]
bull Many sophisticated spinlock realizations were proposed and evaluated
(TS TTS Delay MCS Anderson)
bull Two general types
ndash System spinlocks Kernel OS threads cannot wait Major metrics
atomic operations frequency Shared bus utilization
ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times
RDTEX | Latch internals | RUOUG Seminar
9
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082
THE NEED FOR SPIN
RDTEX | Latch internals | RUOUG Seminar
10
Time to complete CBC latch contention testcase vs number of threads
No spinning _spin_count=0
_spin_count=2000
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182
SPINLOCK REALIZATIONS
RDTEX | Latch internals | RUOUG Seminar
Spinlock Pseudocode Problems
TS
pre-112 mutex
while(Test_and_Set(lock)) Bus saturation
TTS
Oracle latch
while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo
ldquothundering herdsrdquo)
Delay
Latch and 112
Mutex
Adjustable delay Slower under contention
Anderson MCSetc
Queues Widely used in JavaLinux kernel hellip
CPU and memoryoverhead preemptionissues
11
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282
THE TOOLS
RDTEX | Latch internals | RUOUG Seminar12
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382
DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK
DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle
providermodulefunctionname
pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482
ORADEBUG INTERNAL ORACLE DEBUGGER
RDTEX | Latch internals | RUOUG Seminar
bull oradebug call Allows us invoke any internal Oracle function manually
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1
bull oradebug peek Examines contents of any memory location
bull oradebug poke Modifies memory
bull oradebug watch Watch a region of memory
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582
LATCH CONTENTION TESTCASES
RDTEX | Latch internals | RUOUG Seminar
15
create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention
end loop
bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0
bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies
exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip
exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip
bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882
HOW SPINLOCK WORKS
RDTEX | Latch internals | RUOUG Seminar
8
CPU3
Free FreeHolding by CPU 1
Spin polls the spinlock location
Get
Miss Sleep Spin count ( )∆
Holding time (S )
CPU2 Spin Holding the spinlock
Holding by CPU 2
CPU1 Holding the spinlock
The spinlock location
Sleep releases the CPU
bull Oracle latches
ndash Use atomic hardware instruction for Immediate get
ndash If missed process spins by polling latch location during Spin get
ndash Number of spin cycles is bounded by spin count
ndash In spin get not succeed the process acquiring the latch sleeps
hellip
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982
CLASSIC SPINLOCKS
bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip
bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]
bull Many sophisticated spinlock realizations were proposed and evaluated
(TS TTS Delay MCS Anderson)
bull Two general types
ndash System spinlocks Kernel OS threads cannot wait Major metrics
atomic operations frequency Shared bus utilization
ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times
RDTEX | Latch internals | RUOUG Seminar
9
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082
THE NEED FOR SPIN
RDTEX | Latch internals | RUOUG Seminar
10
Time to complete CBC latch contention testcase vs number of threads
No spinning _spin_count=0
_spin_count=2000
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182
SPINLOCK REALIZATIONS
RDTEX | Latch internals | RUOUG Seminar
Spinlock Pseudocode Problems
TS
pre-112 mutex
while(Test_and_Set(lock)) Bus saturation
TTS
Oracle latch
while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo
ldquothundering herdsrdquo)
Delay
Latch and 112
Mutex
Adjustable delay Slower under contention
Anderson MCSetc
Queues Widely used in JavaLinux kernel hellip
CPU and memoryoverhead preemptionissues
11
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282
THE TOOLS
RDTEX | Latch internals | RUOUG Seminar12
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382
DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK
DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle
providermodulefunctionname
pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482
ORADEBUG INTERNAL ORACLE DEBUGGER
RDTEX | Latch internals | RUOUG Seminar
bull oradebug call Allows us invoke any internal Oracle function manually
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1
bull oradebug peek Examines contents of any memory location
bull oradebug poke Modifies memory
bull oradebug watch Watch a region of memory
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582
LATCH CONTENTION TESTCASES
RDTEX | Latch internals | RUOUG Seminar
15
create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention
end loop
bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0
bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies
exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip
exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip
bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982
CLASSIC SPINLOCKS
bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip
bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]
bull Many sophisticated spinlock realizations were proposed and evaluated
(TS TTS Delay MCS Anderson)
bull Two general types
ndash System spinlocks Kernel OS threads cannot wait Major metrics
atomic operations frequency Shared bus utilization
ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times
RDTEX | Latch internals | RUOUG Seminar
9
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082
THE NEED FOR SPIN
RDTEX | Latch internals | RUOUG Seminar
10
Time to complete CBC latch contention testcase vs number of threads
No spinning _spin_count=0
_spin_count=2000
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182
SPINLOCK REALIZATIONS
RDTEX | Latch internals | RUOUG Seminar
Spinlock Pseudocode Problems
TS
pre-112 mutex
while(Test_and_Set(lock)) Bus saturation
TTS
Oracle latch
while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo
ldquothundering herdsrdquo)
Delay
Latch and 112
Mutex
Adjustable delay Slower under contention
Anderson MCSetc
Queues Widely used in JavaLinux kernel hellip
CPU and memoryoverhead preemptionissues
11
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282
THE TOOLS
RDTEX | Latch internals | RUOUG Seminar12
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382
DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK
DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle
providermodulefunctionname
pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482
ORADEBUG INTERNAL ORACLE DEBUGGER
RDTEX | Latch internals | RUOUG Seminar
bull oradebug call Allows us invoke any internal Oracle function manually
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1
bull oradebug peek Examines contents of any memory location
bull oradebug poke Modifies memory
bull oradebug watch Watch a region of memory
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582
LATCH CONTENTION TESTCASES
RDTEX | Latch internals | RUOUG Seminar
15
create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention
end loop
bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0
bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies
exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip
exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip
bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082
THE NEED FOR SPIN
RDTEX | Latch internals | RUOUG Seminar
10
Time to complete CBC latch contention testcase vs number of threads
No spinning _spin_count=0
_spin_count=2000
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182
SPINLOCK REALIZATIONS
RDTEX | Latch internals | RUOUG Seminar
Spinlock Pseudocode Problems
TS
pre-112 mutex
while(Test_and_Set(lock)) Bus saturation
TTS
Oracle latch
while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo
ldquothundering herdsrdquo)
Delay
Latch and 112
Mutex
Adjustable delay Slower under contention
Anderson MCSetc
Queues Widely used in JavaLinux kernel hellip
CPU and memoryoverhead preemptionissues
11
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282
THE TOOLS
RDTEX | Latch internals | RUOUG Seminar12
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382
DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK
DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle
providermodulefunctionname
pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482
ORADEBUG INTERNAL ORACLE DEBUGGER
RDTEX | Latch internals | RUOUG Seminar
bull oradebug call Allows us invoke any internal Oracle function manually
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1
bull oradebug peek Examines contents of any memory location
bull oradebug poke Modifies memory
bull oradebug watch Watch a region of memory
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582
LATCH CONTENTION TESTCASES
RDTEX | Latch internals | RUOUG Seminar
15
create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention
end loop
bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0
bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies
exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip
exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip
bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182
SPINLOCK REALIZATIONS
RDTEX | Latch internals | RUOUG Seminar
Spinlock Pseudocode Problems
TS
pre-112 mutex
while(Test_and_Set(lock)) Bus saturation
TTS
Oracle latch
while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo
ldquothundering herdsrdquo)
Delay
Latch and 112
Mutex
Adjustable delay Slower under contention
Anderson MCSetc
Queues Widely used in JavaLinux kernel hellip
CPU and memoryoverhead preemptionissues
11
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282
THE TOOLS
RDTEX | Latch internals | RUOUG Seminar12
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382
DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK
DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle
providermodulefunctionname
pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482
ORADEBUG INTERNAL ORACLE DEBUGGER
RDTEX | Latch internals | RUOUG Seminar
bull oradebug call Allows us invoke any internal Oracle function manually
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1
bull oradebug peek Examines contents of any memory location
bull oradebug poke Modifies memory
bull oradebug watch Watch a region of memory
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582
LATCH CONTENTION TESTCASES
RDTEX | Latch internals | RUOUG Seminar
15
create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention
end loop
bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0
bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies
exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip
exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip
bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282
THE TOOLS
RDTEX | Latch internals | RUOUG Seminar12
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382
DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK
DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle
providermodulefunctionname
pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482
ORADEBUG INTERNAL ORACLE DEBUGGER
RDTEX | Latch internals | RUOUG Seminar
bull oradebug call Allows us invoke any internal Oracle function manually
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1
bull oradebug peek Examines contents of any memory location
bull oradebug poke Modifies memory
bull oradebug watch Watch a region of memory
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582
LATCH CONTENTION TESTCASES
RDTEX | Latch internals | RUOUG Seminar
15
create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention
end loop
bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0
bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies
exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip
exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip
bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382
DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK
DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle
providermodulefunctionname
pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482
ORADEBUG INTERNAL ORACLE DEBUGGER
RDTEX | Latch internals | RUOUG Seminar
bull oradebug call Allows us invoke any internal Oracle function manually
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1
bull oradebug peek Examines contents of any memory location
bull oradebug poke Modifies memory
bull oradebug watch Watch a region of memory
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582
LATCH CONTENTION TESTCASES
RDTEX | Latch internals | RUOUG Seminar
15
create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention
end loop
bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0
bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies
exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip
exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip
bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482
ORADEBUG INTERNAL ORACLE DEBUGGER
RDTEX | Latch internals | RUOUG Seminar
bull oradebug call Allows us invoke any internal Oracle function manually
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1
bull oradebug peek Examines contents of any memory location
bull oradebug poke Modifies memory
bull oradebug watch Watch a region of memory
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582
LATCH CONTENTION TESTCASES
RDTEX | Latch internals | RUOUG Seminar
15
create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention
end loop
bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0
bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies
exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip
exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip
bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582
LATCH CONTENTION TESTCASES
RDTEX | Latch internals | RUOUG Seminar
15
create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention
end loop
bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0
bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies
exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip
exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip
bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682
ROW CACHE LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
16
Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)
Latch wait
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782
HOW ORACLEREQUESTS THE LATCH
RDTEX | Latch internals | RUOUG Seminar17
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882
DTRACE REVEALS LATCH INTERFACE ROUTINES
Oracle calls the following functions to acquire the latch
ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches
ndash kslgetsl (laddrwaitwhywheremode) - get shared latch
bull 11g - ksl_get_shared_latch (hellip)
ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches
ndash kslgpl(laddrcommentwhywhere) - get parent and all childs
ndash kslfre(laddr) - free the latch
Oracle give us possibility to do the same by oradebug call
RDTEX | Latch internals | RUOUG Seminar
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501
983125983155983141 32983085983138983145983156 983152983151983154983156983155
64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141
18
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982
ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH
To request the latch Oracle kernel routines need
bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition
bull where ndash integer code for location from where the latch is acquired
Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It
contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for
some ldquowhererdquo can be found in x$ksllwksllwlbl
bull mode ndash requesting state for shared lathes
ndash 8 SHARED mode 16 EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
19
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082
LIST OF LATCH ldquoWHERErdquo LOCATIONS
X$KSLLW ndash List of latch [W]here Part of v$latch_misses
ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context
Join this code on indx to
ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder
ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition
ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch
ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by
ldquowhererdquo
RDTEX | Latch internals | RUOUG Seminar
20
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182
V$LATCH_MISSES
bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure
bull The following counters are incremented on each latch sleep
sleep_count (ldquowhererdquo the latch was held)
wtr_slp_count (ldquowhererdquo the latch get was requested)
bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter
bull longhold_count is incremented for shared latches only when
someone held the latch at this ldquowhererdquo for the entire duration ofsleep
RDTEX | Latch internals | RUOUG Seminar
21
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282
LATCH IS HOLDING BY PROCESS NOT SESSION
bull
RDTEX | Latch internals | RUOUG Seminar
983123983156983154983157983139983156 983147983155983157983152983154 983163
991270
983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133
983165
991270983165
983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161
983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154
983123983156983154983157983139983156 983147983155983148983148983156983163
991270
983165
L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155
983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156
Each process has an array of references to the latches it is holding
22
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382
KSLLA IS HIDDEN IN V$PROCESS
bull The process latching info is the kslla structure embedded in theprocess state object
bull Some kslla fields were externalized to SQL
bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a
latch Not externalized directly
bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are
holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to
systematically troubleshoot latch contention
RDTEX | Latch internals | RUOUG Seminar
23
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482
X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET
bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)
bull ksllawat - latch being waited for This is v$processlatchwait
bull ksllawhy ndash ldquowhyrdquo for the latch being waited for
bull ksllawere ndash ldquowhererdquo for the latch being waited for
bull ksllalow ndash bit array of levels of currently holding latches
bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81
bull ksllaps - inter-process post statistics
RDTEX | Latch internals | RUOUG Seminar
24
Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582
THE LATCH STRUCTURE ndash KSLLT
RDTEX | Latch internals | RUOUG Seminar
struct ksllt
ltLatchgt
ldquowhererdquo and ldquowhyrdquo
Level latch class other attributes
Statistics
Latch wait list header
hellip
25
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682
LATCH SIZE BY VERSION
nix 32bit nix 64bit Windows 32bit
734 92 - 120
806 104 - 104817 104 144 104
901 200 160
920 196 240 200
1010 256 208
1020 - 11203 100 160 104
RDTEX | Latch internals | RUOUG Seminar
x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162
F983122983119983117 983160$983147983155983149983142983155983158
983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156
Latch structure was bigger in 101 due to additional latch statistics
26
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782
ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION
bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state
0x00 ndash latch is free
0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7
0x010x02hellip - shared latch holding by 12 hellip processes simultaneously
0x20000000 | pid - shared latch holding exclusively
bull In 11g first latch word shows the pid of the latch holder
0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch
RDTEX | Latch internals | RUOUG Seminar
27
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882
BUSY ORACLE 11G LATCH IN MEMORY
bull 112 Exclusive latch (32-bit platform)
bull 112 Shared latch EXCLUSIVE mode
RDTEX | Latch internals | RUOUG Seminar
SQLgt oradebug peek 200222A0 24
[200222A0 200222B8) = 00000016 00000001 000001D0 00000007
pid^ gets latch level
SQLgt oradebug peek 0xampladdr 24
[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level
983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982
LATCH FIXED TABLES
bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)
bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)
bull In 11g child (allocated on startup) latches are externalized in
v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select
from x$kslltr acquires all parent latches and their children
bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time
ndash kslltwgt hellip - statistics
ndash kslltlvl hellip - attributes
ndash kslltwtt ndash latch wait time
RDTEX | Latch internals | RUOUG Seminar
29
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082
CUMULATIVE V$LATCH STATISTIC COUNTERS
RDTEX | Latch internals | RUOUG Seminar
Statistics x$ksllt When and how it increments
GETS kslltwgt ldquo++rdquo after wait mode latch get
MISSES kslltwff ldquo++rdquo after wait get if it was missed
SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get
SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept
WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get
IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss
IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed
30
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182
MANY STATISTICS WERE LOST IN 102
In 102 Oracle made obsolete many useful latch statistics
Latch size was decreased and the following fields always contain 0now in v$latch
ndash v$latchwaiters_woken
ndash v$latchwaits_holding_latch
ndash v$latchsleep1hellipsleep10
ndash and many undocumented statistics
This was mentioned in 102 Reference
RDTEX | Latch internals | RUOUG Seminar
31
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282
X$KSLLD - LATCH DESCRIPTORS
bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures
bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor
bull Some but not all of kslld fields are externalized via x$kslld andv$latchname
bull Located in the PGA
RDTEX | Latch internals | RUOUG Seminar
32
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382
LATCH ATTRIBUTES
bull Each latch have at least the following attributes in kslldt
ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared
ndash PAR Is the latch Solitary or Parent for the family of child latches
ndash G2C Can two child latches be simultaneously requested in wait mode
using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92
ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above
ndash Level 0-14 Used to prevent latch deadlocks
ndash Class 0-7 Spin and wait class assigned to the latch 92 and above
RDTEX | Latch internals | RUOUG Seminar
33
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482
LATCH ATTRIBUTES BY ORACLE VERSION
RDTEX | Latch internals | RUOUG Seminar
11203 553 154 72 - 6 93
Oracle Number of latches PAR G2C LNG UFS SHARED
7340 53 14 2 3 - -
8063 80 21 7 3 - 38174 152 48 19 4 - 9
9208 242 79 37 - - 19
10202 385 114 55 - 4 47
10203 388 117 58 - 4 48
10204 394 117 59 - 4 50
11106 496 145 67 - 6 81
11107 502 145 67 - 6 83
11201 535 149 70 - 6 86
11202 551 153 72 - 6 91
34
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582
LATCH LEVELS
bull Every latch has a level in the range 0-14 to avoid deadlocks
bull Process can request a latch X in wait mode after obtaining latch Y if and
only if level X gt level Y
bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y
bull If these rules are broken the Oracle process raises
ORA-600[504] - trying to obtain another latch at incompatible level Note 281041
ORA-600[525] - trying to obtain another latch at the same level Note 1388881
ORA-600[526] - latch child order violation Note 1388891
RDTEX | Latch internals | RUOUG Seminar
35
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682
PARENT AND CHILD LATCHES
bull Latch can be a parent child or solitary latch
bull Both parent and child latches share the same latch name
bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss
bull Child latches are allocated during instance startup
bull The parent latch can be gotten independently
bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s
RDTEX | Latch internals | RUOUG Seminar
(latch info) wait_event=0 bits=10
holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl
Context saved from call 0
state=busy
36
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782
LATCH TREES
ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches
Direct SGA access program output for 9206 instance with too smallshared pool
RDTEX | Latch internals | RUOUG Seminar
ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1
waiter ospid 129 sid 72 pid 45
holding a154b7120 library cache (157) level=5 child=17 kglupc child
waiter ospid 18255 sid 65 pid 930
waiter ospid 6690 sid 554 pid 1654
waiter ospid 4685 sid 879 pid 1034
hellip
waiter ospid 29749 sid 180 pid 155
holding a154b7db8 library cache (157) level=5 child=4 kglupc child
waiter ospid 13104 sid 281 pid 220
waiter ospid 24089 sid 565 pid 636
waiter ospid 25002 sid 621 pid 1481
waiter ospid 16930 sid 1046 pid 783hellip
37
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882
LATCH CLASSES
bull Classes with different spin and wait policies
bull Up to 8 classes By default all the latches belong to class 0
bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2
bull Latch can be assigned to class by its number For example
ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000
ndash _LATCH_CLASSES=46
bull Will be discussed later
RDTEX | Latch internals | RUOUG Seminar
SQLgt select from x$ksllclass
INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4
---- ------ ----- -------- ------ ------ ------ ------ ------
0 20000 0 1 8000 8000 8000 8000 8000
hellip
6 100 12 1000 2000 3000 4000 4000 4000
38
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982
LATCH ACQUISITION IN NO-WAIT MODE
bull kslgetl(laddr0whywhere) ndash Exclusive latch get
bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep
ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches
ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches
bull IMMEDIATE_ statistics are updated
bull Used when many candidate latches are available to bypass the
latch level rule or as part of CBC chain examination
RDTEX | Latch internals | RUOUG Seminar
sqlplus nolog demo_03sql39
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082
LATCH WAITS
RDTEX | Latch internals | RUOUG Seminar 40
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182
WAITING FOR THE LATCH
RDTEX | Latch internals | RUOUG Seminar
983123 G A
C983120983125 1 C983120983125 2
983120983154983151983139983141983155983155 A
983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)
983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144
983116983137983156983139983144
983120983154983151983139983141983155983155 B
41
RDTEX | L h i l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282
LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION
bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries
again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of
times
bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep
bull A sleep corresponds to a wait for the latch free wait event
bull Initially a process sleeps for 001 s This time is doubled in every
subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip
RDTEX | Latch internals | RUOUG Seminar
Still did not found any versionwhich used this
May be it was lt= Oracle 6
42
RDTEX | L t h i t l | RUOUG S i
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382
LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81
Latch gets with wait (kslgetl(laddress1hellip))
ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times
ndash Sleep on latch free event with exponential backoff
ndash Repeat
RDTEX | Latch internals | RUOUG Seminar
43
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482
8I LATCH SPIN CODE FLOW USING DTRACE
RDTEX | Latch internals | RUOUG Seminar
bull hellip Event 10046 trace
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1
bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2
983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29
983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3
983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270
sqlplus nolog demo_04_8sql 29
44
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582
POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF
bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec
RDTEX | Latch internals | RUOUG Seminar
12 ]2 )1[(
minus= +wait N
timeout
45
bull Most waits were for nothing ndash latch already was free
bull Latch utilization can not be high
bull Unnecessary spins provoke CPU thrashing
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682
92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE
sqlplus nolog demo_05sql 45
RDTEX | Latch internals | RUOUG Seminar
kslgetl(0x50006318 1)
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt kslges(0x50006318 ) -wait latch get
-gt skgslsgts(0x50006318 ) -spin latch get
-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT
-gt kskthbwt(0x0)
-gt kslwlmod() - set up Wait List
-gt sskgslgf(0x50006318)= 0 -immediate latch get
-gt skgpwwait -sleep latch getsemop(11 17-10 1)
983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140
46
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782
RELIABLE LATCH WAITS
bull Hidden latch wait revolution that we missed Latch relies on waitposting
bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout
bull Exclusive latch spins 20000 cycles by default
bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock
bull Latches assigned to non-default class wait until timeout
bull Version 90 was transient and used mix of backoff and wait posting
This is why _LATCH_WAIT_POSTING parameter disappeared in9i
bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin
RDTEX | Latch internals | RUOUG Seminar
47
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882
IMAGINE THE TIME MICROSCOPE
| a a s | a
Reality One million zoomed time
1 us 1 sec
1 ms 17 min1 sec 115 days
Light speed (300000 kmsec) Sonic speed (300 msec)
CPU tick (2 GHz) 00005 sec
Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec
Max spin time for shared latch (~220005 ticks) ~10 sec
Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec
Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec
OS context switch time (10us-1ms) 10sec-17min
48
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982
LATCH WAITS UNDER TIME MICROSCOPE
Old 8i latch algorithmSpin for latch during 5 seconds
Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min
hellipSpin again for 5 seconds
Sleep again for 23 days (2 sec) hellip
New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)
If spin not succeeded go to sleep until post
According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)
| |
49
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082
_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES
bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and
_LATCH_CLASS_0 parameters
bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000
bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter
bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch
50
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182
TIMED LATCH WAIT POSTING
bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel
bull This can lead to instance hang ndash process will never be woken up
bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout
bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem
bull Generic AIX ports always use timed latch wait posting
bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux
51
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282
FINE GRAIN LATCH TUNING
bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting
bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7
bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo
microseconds using pollsys() (not semtimedop()) system call
ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times
Loop up to ldquoSpinsrdquo cycles
Yield CPU using yield() (or sched_yield())
ndash Sleep for ldquoSleepXrdquo usecs
ndash Then spin again hellip
52
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382
TRACES OF NONDEFAULT CLASS LATCH
983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090
983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086
983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152
991270
9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000
991270 E983158983141983150983156 10046 983156983154983137983139983141
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3
983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482
PERFORMANCE OF LATCH CLASSES
Row cache objects latch contention testcase on ODA (24 SMT)54
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582
IS THE _SPIN_COUNT REALLY STATIC
bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch
waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used
ldquocache buffers chainrdquo latch contention
bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass
(_LATCH_CLASS_X) By default ndash 20000
bull Shared latch spin can be tuned by _SPIN_COUNT value By
default ndash 2000 However this is not trivial
55
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682
SHARED LATCH BEHAVE LIKE ENQUEUE
bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80
bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests
bull This blocking achieved by special 0x40000000 bit in the latch
value This bit is a flag that some incompatible latch acquisitionis in progress
bull Queued processes will be woken up one by one on latch release
bull X mode latch gets effectively serialize the shared latch
bull See my blog for description of experiments
56
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782
TRACE OF SHARED LATCH WAIT
Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the
latch wait list and then spins again
983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141
983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156
983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983147983155983148983159983148983149983151983140()
983147983155983148983155983147983143983155(098316050009BAC )
983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155
983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140
983155983141983149983151983152(27)
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882
SHARED LATCH ACQUISITION
bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned
bull X mode shared latch get spins by default up to 4000 cycles
bull S mode does not spin at all (or spins in unknown way)
S mode get X mode get
Held in S mode Compatible 2_SPIN_COUNT
Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT
58
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982
LATCH RELEASE
bull Free the latch ndash kslfre(laddr)
bull Oracle process releases the latch nonatomically
bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process
bull This requires less bus invalidation and ensures propagation of latch
release to other local cachesbull Not fair policy Spinners on the local CPU board have the
preference
bull Then process posts the first process in the list of waiters
59
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082
LATCH CONTENTION
RDTEX | Latch internals | RUOUG Seminar60
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182
LATCH CONTENTION
bull Occurs when a latch is required by several processes at thesame time
bull Mostly due to bugs or abnormal application behavior
bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp
bull To diagnose the latch contention we need to investigate thelatch statistics [8]
61
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282
DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS
Latch requests arrival rate
Immediate gets efficiency
Latch sleeps ratio
Latch wait time per second
Latch spin efficiencymisses
getsspin∆
∆= _σ
misses
sleepsK
∆
∆=
time
gets
∆
∆=λ
gets
misses
∆
∆= ρ
time
timewait W
∆
∆=
_
Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics
62
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382
SAMPLED AVERAGES
By sampling of x$ksupr we can estimate
ndash Length of latch wait queue
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS
ndash Number of spinning processes in 10g
SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS
Sampling of v$lathholder estimates latch utilization
Using DTrace we can measure
ndash Latch acquisition and holding times
ndash Actual spin count
w L
s N
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482
DERIVED LATCH STATISTICS
Latch utilization (PASTA)
Average holding time
Length of latch wait list
Recurrent sleeps ratio
Latch acquisition time
time
timeholdinglatchU
∆
∆=asymp
__ ρ
Re_100
___
questsGet
TimeSnap MissGet Pct S
lowast==
λ
W L =
κ
κ σ 1minus+
)(1
W N T saq += minusλ
64
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582
LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154
0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150
983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162
M983145983155983155 983143983141983156 983154983144983151= 022
983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013
983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28
983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021
983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72
983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013
983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002
A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139
983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139
L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150
983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------
2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |
8192 |8192 |8192 |8192 |
16384 |16384 |16384 |16384 |
32768 |32768 |32768 |32768 |
65536 |65536 |65536 |65536 |
nsnsnsns
A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139
983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682
LATCH CONTENTION DIAGNOSTICS IN 92-11G
bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section
bull Look for the latch with highest W
bull Symptoms of contention for the latch
ndash W gt 1 secsec
ndash Utilization gt 10
ndash Acquisition (or sleeping) time sufficiently greater then holding time
bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics
bull The script and v$latch_misses reveal ldquowhererdquo the contention arise
ρ
66
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782
LATCH LEVELS AND LATCH CONTENTION
bull Process holding latch can only acquire latch with higher level
bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches
bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention
bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms
bull Use latch_treesql script to confirm this scenario
67
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882
BEWARE OF CERTAIN V$X$
bull Frequent scans of some x$ tables induce latch contention
ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation
ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g
ndash x$ksmsp ndash shared pool latch
ndash x$kslltr (v$latch) ndash all parent latches in 11g
ndash hellip
68
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982
TREATING THE LATCH CONTENTION
bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip
bull Many brilliant books exist on this topic Out of scope for thispresentation
bull Install latest Oracle PSU and OS recommended patches Severallatch related examples
ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value
ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed
in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch
PHKL_38397 or later
69
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082
WHEN WE NEED TO TUNE SPIN COUNT
bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite
bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources
bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For
example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch
upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application
bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption
70
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182
TUNING THE SPIN COUNT EFFICIENTLY
bull Beware of side effects You should have enough free CPU
bull First diagnose the root cause of latch contention Measure the
latch statistics install relevant patches
bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range
bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change
bull Spin count adjustment is different for exclusive and shared latches
bull Nonstandard latch classes may decrease CPU consumption
bull It is a common myth that CPU time will raise infinitely while weincrease spin count
71
SPIN COUNT ADJUSTMENT
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282
SPIN COUNT ADJUSTMENT
Shared latches
bull Dynamically by _SPIN_COUNT parameter
bull Good starting point is the multiple of default 2000 value
bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart
Exclusive latches
bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart
bull Good starting point is the multiple of default 20000 value
bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my
blog for related mathematics
72
SPIN SCALING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382
SPIN SCALING
bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on
K = sleeps ratio=ldquoAvg SlpsMissrdquo
bull If spin is inefficient then
Doubling the spin count will reduce sleeps ratio by 2 and doubles
the spin CPU consumption
bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then
Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by
If the spin is already efficient it is worth to increase the spin count
)1( K +
10geK
10ltltK
I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150
10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10
983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161
73
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482
CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE
L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124
74
LONG LATCH HOLDING TIME CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582
LONG LATCH HOLDING TIME CPU THRASHING
bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU
bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch
bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls
bull Transition to this metastable state is more likely if workload of
your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU
scheduling scale [7]
75
CPU THRASHING
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682
CPU THRASHING
bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself
All wait times will be inflated by wait for CPU However latchholding time should be in normal range
bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol
bull To prevent the priority inversion use
ndash FX priority class on Solaris
ndash HPUX_SCHED_NOAGE on HPUX
ndash SCHED_BATCH policy on Linux
76
LATCH SMP SCALABILITY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782
LATCH SMP SCALABILITY
bull If latch utilization is in single CPU environment
bull Then in N CPU server latch utilization will be This can be
problematic
ndash If single CPU system held latches only for 1 of time
ndash 48 CPU server with the same per-CPU load will hold latches for 50
ndash 128 CPU Cores server will suffer huge latch (and mutex) contention
bull This is also known as Software lockout It may substantially affect
contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions
1 ρ
1 ρ N N asymp
77
BIBLIOGRAPHY
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882
BIBLIOGRAPHY
1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999
2 Tanel Poder blog httpblogtanelpodercom
3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011
4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965
5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared
Memory Systems Proc of 29th Hawaii Conference on System Sciences1996
7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g
httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking
Proc of 5th Workshop on Data Management on New Hardware 2009
9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011
78
QA
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982
QA
bull Questions
bull Comments
79
ACKNOWLEDGEMENTS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082
ACKNOWLEDGEMENTS
bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my
investigationsbull Thanks to my colleagues for discussions
bull Thanks to all our customers for participating in latch
troubleshooting
80
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182
A983150983140983154983141983161 N983145983147983151983148983137983141983158
983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149
A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157
983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137
983159983159983159983154983140983156983141983160983154983157
THANK YOU
RDTEX | Latch internals | RUOUG
Seminar
IN MEMORY OF MY FRIEND FOR 30 YEARS
RDTEX | Latch internals | RUOUG Seminar
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82
7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug
httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282
A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)
983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089
82