b10: a new spin on some old latches richard banville fellow

B10: A New Spin on Some Old Latches

Richard BanvilleFellow

© 2008 Progress Software Corporation2

Agenda

Monitoring Latch usage Performance


Some Definitions First

What is a “latch”• Synchronization mechanism

• Latches vs “locks”

What is a critical section• Code access to shared resource

• Types of resources

How is a latch obtained• test, test-and-set, spin loop

– Performed on “shared” value

Single threading access


Latch Info

VST access to latch information

_Latch-Type Latch type

_Latch-Name Name

_Latch-Lock # times locked

_Latch-Wait Nap count

_Latch-HoldUser # of last holder

_Latch-QholdUser # of last holder of queue latch

for each _latch:


_Latch-Type: MT_LT_QUEUE

Queue Latch governor

LatchSemaphore

• Multi level latch

• Wakeup order guaranteed

• -spin 0 forces all latches

Test, test-and-set spin loop


_Latch-Type: MT_LT_SPIN

Queue Latch governor

LatchSemaphore

• Multi level latch

• Wakeup order guaranteed

• -spin 0 forces all latches

Spin Latch Latch

Short Nap

• Single level latch

• Most performant

“backoff”



-nap to -napmax


Latch Implementations

Spin or queue latch acts as governor of “mux” latch table

governor

Resources protectedMux latch table (128)

• Multi level latch with one governor

• Associates critical section w/resource

• Underlying one to many relationshipTurns 1 latch into 128 latches

“Mux” latch

request



Spin or queue latch acts as governor of object latchResources protected

• Multi level latch with one governor


• Underlying one to one relationshipTurns 1 latch into many latches

governor

• Enhanced mux latch

Object latch - type 1

request



Spin or queue latch maintained within the resource itselfResources protected

• Single level latch. No governor


• One to one relationship

• Greatly improves concurrency

No single access funneling

• Statistics maintained as if 1 latch exists

Object latch - type 2

request



Table of spin or queue latchsResources protectedLatch table

• Single level latch with no governor


• Underlying one to many relationship

Latch families

request

• Statistics typically make this look like one latch

• Modified “mux” latch mechanism


Latch Holder

_Latch-QHolder• Last owner of queue type latch

• Not last queued user

_Latch-Holder• Last owner of spin latch

• Not honored for “true” object latches– Type 2

Who am I waiting on?


Latch Counts

_Latch-Lock• # times latch locked/acquired

• High numbers are OK

_Latch-Wait• # times user had to wait (after spin)

• High numbers may not be good– …but is not wrong– Performance issue potential

• Balance with –spin value (see tuning)

Activity statistics (or lack there of)


Latch Names

_Latch-Name

LKP LKF LHT LHT2 LHT3 LHT4 SEQ PWQ AIW BIW

BFP BHT BF1 BF2 BF3 BF4 BF5 BF6 BF7 CPQ

LRU LR2 LR3 LR4 L27 L28 L28 L30 L31

MTX USR OM AIB BIB SCC GST TXT TXQ


Agenda



Miscellaneous Latches

USR*• Protects login (user) control structures

SCC*• Protects schema locking operation

GST• Protects shared memory allocation

*Indicates queue latch


Obsolete Latches

Removed• AIW, LR2, LR3, LR4

Not in use• BIW

Renamed & reserved for future use• BF5 L27

• BF6 L28

• BF7 L29

• BF8 L30

Latches removed or reserved


Latches Protecting

Database Update Actions


Update Latches - MTX

DB Update ActionDB Update Action Record BI noteRecord BI note Record AI noteRecord AI note Perform ActionPerform Action



Top of the food chain – busiest/longest update latch

• Protects txn allocation & bi/ai note order

• Quiet point maintenance

• Online backup startup

Latches protecting recovery logging procedure

• High activity

• High nap rate

MTX

Not allowed

User 1

User 2

User 3


BIB and AIB latch usage

-bibufs 10

Free(a)

Free(b)

Free(c)

Free(d)

Free(e)

32 31

30

29

Modified QueueFree List

BI

Current Output Buffer

New Notes (Actions)

Forward Processing

Bi (ai) buffer life cycle

B I W

Very little contention

until BIW active

OLTP latch order:• MTX, BIB, AIB

Page writer latch order:• BIB or AIB

BIB

BIB

BIB

User


Latches – LRU & PWQ

3232 20482048 10241024

-B least recently used (LRU) chain maintenance

. . .. . .81928192 10561056LRU MRU

High activityHigh nap rate

LRU


Latches – LRU & PWQ

3232 20482048 10241024


. . .. . .81928192 10561056LRU MRU

10881088

20802080

32003200

Page writer queue (PWQ)

myDBmyDB

PageWriter

PageWriter

Dirty buffer

Very little contentionunless too many APWs

LRU

PWQ

PWQ

LRU

UserUser


Latches – LRU & CPQ

3232 20482048 10241024


. . .. . .81928192 10561056LRU MRU

10881088

20802080

32003200

Page writer queue (PWQ)

myDBmyDB

PageWriter

PageWriter

10241024

71047104

60806080

Check point queue (CPQ)

Check-point

Check-point

Very little contention

CPQ

CPQ


Online backup queue - BFP

Online “point in time” backup

User 1

BI

Area 6 Area 7 Area 8 …

Online Backup

.bak

Backupqueue

Modify requestModify request

• Very little contention

• Only used w/online backup

Back it up and mark it

BFP

TXEMTX

BFP


Update Latches

TXT• Protects transaction table modifications

TXQ• Protects acquire/release of transaction end

lock (TXE)

SEQ• Protects sequence control structures

Moderate activity Little contention

Latches for write operation structures


Latches Protecting

Data Access Actions

and Housekeeping


Disk Storage Files

Storage Areas

Data Area

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ta

ble

A

Ta

ble

B

Cu

st

Tb

l

Ta

ble

D

Ind

ex

A-1

Ind

ex

B-1

Na

me

Id

x

Cu

st#

Id

x

SR

ep

Id

x

Ind

ex

D-1

Ind

ex

D-2

Logical

Physical


Disk Storage Files

Storage Areas

Schema Area Area 7 Area 8

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ta

ble

A

Ta

ble

B

Cu

st

Tb

l

Ta

ble

D

Ind

ex

A-1

Ind

ex

B-1

Na

me

Id

x

Cu

st#

Id

x

SR

ep

Id

x

Ind

ex

D-1

Ind

ex

D-2

Logical

Physical

Lan

gu

ages

Sto

rage E

ng

ine


Disk Storage Files

Storage Areas and Object Mapping

Schema Area Area 7 Area 8

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ta

ble

A

Ta

ble

B

Cu

st

Tb

l

Ta

ble

D

Ind

ex

A-1

Ind

ex

B-1

Na

me

Id

x

Cu

st#

Id

x

SR

ep

Id

x

Ind

ex

D-1

Ind

ex

D-2

Logical

Physical

Mapping

Lan

gu

ages

Sto

rage E

ng

ine


Object Cache – OM Latch

Ta

ble

A

Ta

ble

B

Cu

st

Tb

l

Ta

ble

D

Ind

ex

A-1

Ind

ex

B-1

Na

me

Id

x

Cu

st#

Id

x

SR

ep

Id

x

Ind

ex

D-1

Ind

ex

D-2

Logical

Mapping

Lan

gu

ages

-omsize

OM cache loaded on demand

Very high activityHigh nap rate

OM latch needed for

paging maintenance

Get _StorageObject record

Schema

OM


Object Cache - OM Latch

Ta

ble

A

Ta

ble

B

Cu

st

Tb

l

Ta

ble

D

Ind

ex

A-1

Ind

ex

B-1

Na

me

Id

x

Cu

st#

Id

x

SR

ep

Id

x

Ind

ex

D-1

Ind

ex

D-2

Logical

Mapping

Lan

gu

ages

OM cache loaded at startup

No latching, no paging

Zero contention! Primary CachePrimary Cache

Little contention 2nd cache for over flow &

OM latch needed for

paging maintenance

Get _StorageObject record

Schema

Secondary CacheSecondary Cache new objects

OM

Schema

-omsize


Buffer Pool Hash Table - BHT

Hash Table (-hash)

Shared memory block lookup

List of (–B) buffer pool entries(unordered)

Buffer pool location lookup single threaded

User 1

User 2

User 3

User 4

Block ID

Block ID

Block ID

Block ID

BHT

High activity, few naps

Ptr to buffer


Buffer Pool Hash Table - BHT

Hash Table (-hash)

BHT now latch family of 256


Buffer pool location lookup multi-threaded

Block IDUser 1

User 2

User 3

User 4

Block ID

Block ID

Block ID

High activity, few naps

Ptr to buffer

BHT

BHT

BHT

. . .

BHT

BHT


-B buffer pool entry info – BF[1-4]

Database access latches


Ptr to buffer

Hash Table (-hash)

BHT

BHT

BHT

. . .

BHT

BHT




Buffer pool info structure supports

Hash Table (-hash)

Database access latches

max of 4 concurrent threads.

BF1

BF2

BF3

BF4

Latch family of 4Ptr to buffer

BHT

BHT

BHT

. . .

BHT

BHT




Block IDUser 1

User 2

User 3

User 4

Block ID

Block ID

Block ID

BF latch family of 4 Object latch type 2Ptr to buffer

Hash Table (-hash)

BHT

BHT

BHT

. . .

BHT

BHT



Hash Table (-hash)List of (–B) buffer pool entries(unordered)

Block IDUser 1

User 2

User 3

User 4

Block ID

Block ID

Block ID

• -B info structure supports –B threads

• 4 latch slots used for statistics only

Ptr to bufferBF latch family of 4 Object latch type 2

BHT

BHT

BHT

. . .

BHT

BHT


Lock Table Latches (LKF, LKP & LHT)

Free Chain (-L entries)

Hash Chain Anchor Table-lkhash entries

Acquire a lock

LKF

Plain latchSingle threaded accessHigh activityLow nap rate





Protected by LHT (LKT)

• Record locks• Record “get” locks

• Table locks• Purged record locks

and LKP latches

Type 1 object latchSingle threaded latch access

Multi threaded chain access

LHT

LKP

LHT

LKP

• High activity/waits

• Low activity/waits


LHT1

LHT2

LHT3

LHT4




Protected by LHT[1-4]

• Record locks

• Record “get” locks

Type 1 object latch family

Multi threaded latch access

Multi threaded chain access

- Added 4 governors

Enhanced concurrency

• Low contention/waits

LKP

• High activity, few waits


Agenda



Tuning parameters

-spin• # retries before nap

– 0 or 1 or 10,000 or 6,000 * # cpus or …• Can be changed online

-mux• Use multiplexing/governor (1)

-nap• Initial amount to nap (10 ms)

-napmax*• Max amount to nap (5 sec)

-napinc and -napstep• Both obsolete and ignored

Nothing new here


Foresight and insight

CPU usage• -spin ~6,000 * # CPUs

• CPU thrashes, decrease –spin

• > 16 CPUs, avoid cache line ping pong

• Activity matters

Activity• # latches acquired

• # latch waits

• Application throughput

What to monitor


Performance – 10.1C vs 10.1B

0

200,000

400,000

600,000

800,000

1,000,000

OM

LHT

LKF

LRU

BH

T

BF1

BF2

BF3

BF4

Readprobe Latch Results

10.1c10.1b

House Keeping Data Access


Performance – 10.1C vs 10.1B

0

100,000

200,000

300,000

400,000

500,000

OM

LHT

LKF

LRU

BH

T

BF1

BF2

BF3

BF4

Readprobe Latch Results

10.1c10.1b

Housekeeping Data Access

Latches/sec

LRU 31% Buf 43%


Performance – wait for it

0

10,000

20,000

30,000

40,000

50,000

Readprobe Latch Waits (per sec*)

10.1c10.1b

10.1c 0 0 0 45,023 10 1 1 1 2

10.1b 298 31,141 8,699 7,790 969 75 100 172 88

OM LHT LKF LRU BHT BF1 BF2 BF3 BF4

4,294 (~9%) fewer latch waits/sec in 10.1c


Readprobe Data Access Results

50,000

100,000

150,000

200,000

250,000

300,000

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101

Rec

ord

s R

ead

10.1C

10.1B

Performance – you got it!

~40%~56%

0 4 8 12 16 24 32 256 512 768 992# Users


In Summary

Better insight We’ve done some things You can too

• Tuning

• Move to 10.1C


Questions?


Some Questions Answered

What does MTX stand for• Micro transaction – It was the first latch

implemented in Progress(V5) and was only held after the database was locked. It maintained database consistency between multi-block operations. Today it is used for serializing operations. The TXE is used to maintain database consistenct between multi-block operations.


Thank You

b10: a new spin on some old latches richard banville fellow

Documents

latchtype latch type

latch implementationsspin

latch holder

latch information

latch counts

queue latch acts

latch infovst access

mux latch mechanism