high performance cache replacement using re-reference interval prediction (rrip) aamer jaleel, kevin...
TRANSCRIPT
High Performance Cache Replacement Using
Re-Reference Interval Prediction (RRIP)
Aamer Jaleel, Kevin Theobald,
Simon Steely Jr., Joel Emer
Intel Corporation, VSSAD
International Symposium on Computer Architecture ( ISCA – 2010 )
Motivation
• Factors making caching important• Increasing ratio of CPU speed to memory speed• Multi-core poses challenges on better shared cache
management
• LRU has been the standard replacement policy at LLC• However LRU has problems!
22
Problems with LRU Replacement
33
Working set larger than the cache causes thrashing
References to non-temporal data (scans) discards frequently referenced working set
miss miss miss missmiss
hit hit hit miss hit hitmiss missscan scan scan
Wsize
Wsize
LLCsize
Our studies show that scans occur frequently in many commercial workloads
hit hit hit hit hit
Desired Behavior from Cache Replacement
44
mis
s
mis
s
mis
s
mis
s
mis
s
Working set larger than the cache Preserve some of working set in the cache
Recurring scans Preserve frequently referenced working set in the cache
hit hit hit hit hitscan scan scanhit hit hit
Wsize
LLCsize
Prior Solutions to Enhance Cache Replacement
55
Dynamic Insertion Policy (DIP) Thrash-resistance with minimal changes to HW
GOAL: Design a High Performing Scan-Resistant Policy that Requires Minimum Changes to HW
Least Frequently Used (LFU) addresses scans
LFU adds complexity and also performs bad for recency friendly workloads
Working set larger than the cache Preserve some of working set in the cache
Recurring scans Preserve frequently referenced working set in the cache
Belady’s Optimal (OPT) Replacement Policy
• Replacement decisions using perfect knowledge of future reference order
• Victim Selection Policy: • Replaces block that will be re-referenced furthest in future
66
Cache Tag a c b h f d g e
4 13 11 5 3 6 9 1“Time” when block will be referenced
next
0 1 2 3 4 5 6 7Physical Way #
victim block
Practical Cache Replacement Policies
• Replacement decisions made by predicting the future reference order
• Victim Selection Policy: • Replace block predicted to be re-referenced furthest in future
• Continually update predictions on the future reference order• Natural update opportunities are on cache fills and cache hits
77
Cache Tag a c b h f d g e
0 1 2 3 4 5 6 7Physical Way #
victim block
4 13 11 5 3 6 9 1“Predicted Time” when block
will be referenced next~ ~ ~ ~ ~ ~ ~ ~
LRU Replacement in Prediction Framework
88
• The “LRU chain” maintains the re-reference prediction
• Head of chain (i.e. MRU position) predicted to be re-referenced soon
• Tail of chain (i.e. LRU position) predicted to re-referenced far in the future
• LRU predicts that blocks are re-referenced in reverse order of reference
• Rename “LRU Chain” to the “Re-Reference Prediction (RRP) Chain ”
• Rename “MRU position” RRP Head and “LRU position” RRP Tail
h g f e d c b a
MRU position LRU positionRRP head RRP tail
0 1 2 3 4 5 6 7LRU chain
position stored with each cache
block
Practicality of Chain Based Replacement
• Problem: Chain based replacement is too expensive!• log2(associativity) bits required per cache block (16-way requires 4-bits/block)
• Solution: LRU chain positions can be quantized into different buckets• Each bucket corresponds to a predicted Re-Reference Interval• Value of bucket is called the Re-Reference Prediction Value (RRPV)
• Hardware Cost: ‘n’ bits per block [ ideally you would like n < log2A ]
99
RRP Head RRP Tail
RRPV (n=2): 3
‘distant’
2
‘far’
1
‘intermediate’
0‘near-
immediate’Qualitative Prediction:
h g f e d c b a
Representation of Quantized Replacement (n = 2)
1010
Cache Tag
3
a
2
c
3
b
0
h
1
f
1
d
0
g
1
e
RRPV
0 1 2 3 4 5 6 7Physical Way #
RRP Head RRP Tail
RRPV: 3
‘distant’
2
‘far’
1
‘intermediate’
0‘near-
immediate’Qualitative Prediction:
h g f e d c b a
Emulating LRU with Quantized Buckets (n=2)
• Victim Selection Policy: Evict block with distant RRPV (i.e. 2n-1 = ‘3’) • If no distant RRPV (i.e. ‘3’) found, increment all RRPVs and repeat the search• If multiple found, need tie breaker. Let us always start search from physical
way ‘0’
• Insertion Policy: Insert new block with RRPV=‘0’
• Update Policy: Cache hits update the block’s RRPV=‘0’
1111
Cache Tag
3
a
2
c
3
b
0
h
1
f
1
d
0
g
1
e
RRPV
0 1 2 3 4 5 6 7Physical Way #
victim block
0
s
hit
0
But We Want to do BETTER than LRU!!!
Re-Reference Interval Prediction (RRIP)
• Framework enables re-reference predictions to be tuned at insertion/update• Unlike LRU, can use non-zero RRPV on insertion• Unlike LRU, can use a non-zero RRPV on cache hits
• Static Re-Reference Interval Prediction (SRRIP)• Determine best insertion/update prediction using profiling [and apply to all apps]
• Dynamic Re-Reference Interval Prediction (DRRIP)• Dynamically determine best re-reference prediction at insertion
1212
Cache Tag
3
a
2
c
3
b
0
h
1
f
1
d
0
g
1
e
RRPV
0 1 2 3 4 5 6 7Physical Way #
Static RRIP Insertion Policy – Learn Block’s Re-reference Interval
• Key Idea: Do not give new blocks too much (or too little) time in the cache• Predict new cache block will not be re-referenced soon• Insert new block with some RRPV other than ‘0’• Similar to inserting in the “middle” of the RRP chain
• However it is NOT identical to a fixed insertion position on RRP chain (see paper)
1313
Cache Tag
3
a
2
c
3
b
0
h
1
f
1
d
0
g
1
e
RRPV
0 1 2 3 4 5 6 7Physical Way #
victim block
2
s
2
Static RRIP Update Policy on Cache Hits
• Hit Priority (HP)• Like LRU, Always update RRPV=0 on cache hits. • Intuition: Predicts that blocks receiving hits after insertion will be re-referenced
soon
1414
Cache Tag
2
s c
3
b
0
h
1
f
1
d
0
g
1
e
RRPV
0 1 2 3 4 5 6 7Physical Way #
hit
0
An Alternative Update Scheme Also Described in Paper
SRRIP Hit Priority Sensitivity to Cache Insertion Prediction at LLC
1515
n=1
Averaged Across PC Games, Multimedia, Server, and SPEC06 Workloads on 16-way 2MB LLC
Re-Reference Interval Prediction (RRIP) Value At Insertion
n=1 is in fact the NRU replacement policy commonly used in commercial processors
SRRIP Hit Priority Sensitivity to Cache Insertion Prediction at LLC
1616
n=1 n=2 n=3 n=4 n=5
Averaged Across PC Games, Multimedia, Server, and SPEC06 Workloads on 16-way 2MB LLC
Regardless of ‘n’ Static RRIP Performs Best When RRPVinsertion is 2n-2Regardless of ‘n’ Static RRIP Performs Worst When RRPVinsertion is 2n-1
Re-Reference Interval Prediction (RRIP) Value At Insertion
Why Does RRPVinsertion of 2n-2 Work Best for SRRIP?
• Before scan, re-reference prediction of active working set is ‘0’
• Recall, NRU (n=1) is not scan-resistant• For scan resistance RRPVinsertion MUST be different from RRPV of working set blocks
• Larger insertion RRPV tolerates larger scans• Maximum insertion prediction (i.e. 2n-2) works best!
• In general, re-references after scan hit IF
Slen < ( RRPVinsertion – Starting-RRPVworkingset) * (LLCsize – Wsize)
SRRIP is Scan Resistant for Slen < ( RRPVinsertion ) * (LLCsize – Wsize)
1717
hit hit hit ? hit hit? ?scan scan scan
Slen
Wsize
For n > 1 Static RRIP is Scan Resistant! What about Thrash Resistance?
DRRIP: Extending Scan-Resistant SRRIP to Be Thrash-Resistant
1818
miss miss miss missmiss
• Always using same prediction for all insertions will thrashes the cache
• Like DIP, need to preserve some fraction of working set in cache• Extend DIP to SRRIP to provide thrash resistance
• Dynamic Re-Reference Interval Prediction:• Dynamically select between inserting blocks with 2n-1 and 2n-2 using Set Dueling• Inserting blocks with 2n-1 is same as “no update on insertion”
SRRIP
miss hit hit hit hit DRRIP miss miss miss miss
DRRIP Provides Both Scan-Resistance and Thrash-Resistance
Performance Comparison of Replacement Policies
1919
Static RRIP Always Outperforms LRU Replacement Dynamic RRIP Further Improves Performance of Static RRIP
16-way 2MB LLC
Cache Replacement Competition (CRC) Results
2020
16-way 1MB Private Cache
65 Single-Threaded Workloads
16-way 4MB Shared Cache
165 4-core Workloads
Averaged Across PC Games, Multimedia, Enterprise Server, SPEC CPU2006 Workloads
DRRIP
DRRIP
Un-tuned DRRIP Would Be Ranked 2nd and is within 1% of CRC WinnerUnlike CRC Winner, DRRIP Does Not Require Any Changes to Cache Structure
Total Storage Overhead (16-way Set Associative Cache)
2121
• LRU: 4-bits / cache block
• NRU 1-bit / cache block
• DRRIP-3: 3-bits / cache block
• CRC Winner: ~8-bits / cache block
DRRIP Outperforms LRU With Less Storage Than LRU
NRU Can Be Easily Extended to Realize DRRIP!
Summary
• Scan-resistance is an important problem in commercial workloads• State-of-the art policies do not address scan-resistance
• We Propose a Simple and Practical Replacement Policy• Static RRIP (SRRIP) for scan-resistance• Dynamic RRIP (DRRIP) for thrash-resistance and scan-resistance
• DRRIP requires ONLY 3-bits per block• In fact it incurs less storage than LRU
• Un-tuned DRRIP would be 2nd place in CRC Championship• DRRIP requires significantly less storage than CRC winner
2222
2323
Q&A
2424
Q&A
2525
Q&A
Static RRIP with n=1
• Static RRIP with n = 1 is the commonly used NRU policy (polarity inverted)• Victim Selection Policy: Evict block with RRPV=‘1’ • Insertion Policy: Insert new block with RRPV=‘0’• Update Policy: Cache hits update the block’s RRPV=‘0’
2626
Cache Tag
1
a
1
c
1
b
0
h
1
f
1
d
0
g
1
e
RRPV
0 1 2 3 4 5 6 7Physical Way #
victim block
0
s
hit
0
But NRU Is Not Scan-Resistant
2
SRRIP Update Policy on Cache Hits
1
• Frequency Priority (FP): • Improve re-reference prediction to be shorter than before on hits (i.e. RRPV--). • Intuition: Like LFU, predicts that frequently referenced blocks should have
higher priority to stay in cache
2727
Cache Tag
2
s c
3
b
0
h
1
f
1
d
0
g
1
e
RRPV
0 1 2 3 4 5 6 7Physical Way #
SRRIP-HP and SRRIP-FP Cache Performance
2828
-10
-5
0
5
10
15
20
GAMES MULTIMEDIA SERVER SPEC06 ALL
% F
ewer
Cac
he M
isse
s Re
lativ
e to
LRU
n=1 n=2 n=3 n=4 n=5
-10
-5
0
5
10
15
20
GAMES MULTIMEDIA SERVER SPEC06 ALL
% F
ewer
Cac
he M
isse
s Re
lativ
e to
LRU
n=1 n=2 n=3 n=4 n=5
SRRIP-Frequency Priority
SRRIP-Hit Priority
• SRRIP-HP has 2X better cache performance relative to LRU than SRRIP-FP
• We do not need to precisely detect frequently referenced blocks
• We need to preserve blocks that receive hits
Common Access Patterns in Workloads
• Stack Access Pattern: (a1, a2,…ak,…a2, a1)A
• Solution: For any ‘k’, LRU performs well for such access patterns
• Streaming Access Pattern: (a1, a2,… ak) for k >> assoc• No Solution: Cache replacement can not solve this problem
• Thrashing Access Pattern: (a1, a2,… ak)A , for k > assoc• LRU receives no cache hits due to cache thrashing• Solution: preserve some fraction of working set in cache (e.g. Use BIP)
• BIP does NOT update replacement state for the majority of cache insertions
• Mixed Access Pattern: [(a1, a2,…ak,…a2, a1)A (b1, b2,… bm)] N, m > assoc-k
• LRU always misses on frequently referenced: (a1, a2, … ak, … a2, a1)A
• (b1, b2, … bm) commonly referenced to as a scan in literature
• In absence of scan, LRU performs well for such access patterns• Solution: preserve frequently referenced working set in cache (e.g. use LFU)
• LFU replaces infrequently referenced blocks in the presence of frequently referenced blocks
2929
Gam
es,
Mu
ltim
ed
ia,
En
terp
rise
Serv
er,
Mix
ed
Work
load
s
-100
10203040506070
halfl
ife2
halo
gunm
etal
2
final
-fant
asy
phot
osho
p
rend
erm
an sap
tpc-
c
app-
serv
er
cact
usAD
M_b
sphi
nx3_
a
hmm
er_n
mcf
_r
bzip
2_c
GAM
ES
MU
LTIM
EDIA
SERV
ER
SPEC
06 ALL%
Few
er M
isse
s Re
lativ
e to
LRU
DIP
HYB(LRU, LFU)
Performance of Hybrid Replacement Policies at LLC
• DIP addresses SPEC workloads but NOT PC games & multimedia workloads
• Real world workloads prefer scan-resistance instead of thrash-resistance 3030
PC Games / multimedia SPEC CPU2006server Average
4-way OoO Processor, 32KB L1, 256KB L2, 2MB LLC
Understanding LRU Enhancements in the Prediction Framework
• Recent policies, e.g., DIP, say “Insert new blocks at the ‘LRU position’”• What does it mean to insert an MRU line in the LRU position? • Prediction that new block will be re-referenced later than existing blocks in the
cache• What DIP really means is “Insert new blocks at the `RRIP Tail’ ”
• Other policies, e.g., PIPP, say “Insert new blocks in ‘middle of the LRU chain’”• Prediction that new block will be re-referenced at an intermediate time
3131
RRP Head RRP Tail
The Re-Reference Prediction Framework Helps Describe
the Intuitions Behind Existing Replacement Policy Enhancements
h g f e d c b a
Performance Comparison of Replacement Policies
3232
Static RRIP Always Outperforms LRU Replacement Dynamic RRIP Further Improves Performance of Static RRIP
16-way 2MB LLC