hbase application performance improvement
DESCRIPTION
Changes which can be made to get the best read performance out of HBase.TRANSCRIPT
HBase Caching & Performance
Biju Nair 11 June 2014
Mo<va<on
• Efficient HBase cluster capacity u<liza<on – 32 cores per node – 128 GB of memory per node – SSD storage in all nodes
• Eliminate/Reduce spikes in HBase “get” calls • Meet “get” query performance requirements – Under 10 ms for 99% of queries – Median latency 2 to 3 ms – 99.99% latency under 50 ms
2
Data and Query
• Time series data – Tickers and aXributes – Monthly data stored in a column; 256 bytes – Up to 20 years worth of data
• Queries – “get”s for up to 1 year data; 3072 bytes
3
HBase Internals (Simplified)
HBase Memory (RS)
Mem Store
Block cache
HBase Storage
WAL Store Files
4
HBase Write Path (Simplified)
HBase Memory (RS)
Mem Store
Block cache
HBase Storage
WAL Store Files
1
2
5
HBase Read Path (Simplified)
HBase Memory (RS)
Mem Store
Block cache
HBase Storage
WAL Store Files
1
2
6
Baseline Test Observa<ons
• Java GC causing spikes in response <mes – Spikes were pronounced with increased heap size
• Less than 10% u<liza<on of node CPUs • Less than 15% u<liza<on of node memory – JVM heap size inefficient beyond 16 GB
• Block cache u<liza<on was inefficient – Low hit ra<o and high evic<on rates
7
Proposed Changes
• Use off-‐heap caching – Minimize spikes in response <me due to GC
– Increased u<liza<on of node memory
• Decrease table block size • Run more region servers per node
8
HBase Off-‐Heap Caching
HBase Memory (RS)
Mem Store
Block cache (L1) Idx & BF data
HBase Storage
WAL Store Files
Off-‐heap cache (L2) Tbl Data (Bucket Cache)
9
HBase Read Path (Simplified)
HBase Memory (RS)
Mem Store
Block cache
HBase Storage
WAL Store Files
1
2
L2 Cache
3
4
10
Bucket Cache Configura<on
• Hbase env.sh HBASE_REGIONSERVER_OPTS parameters – Xmx – XX:MaxDirectMemorySize
• Hbase site.xml properties – hbase.regionserver.global.memstore.upperLimit – hfile.block.cache.size – hbase.bucketcache.size – hbase.bucketcache.ioengine – hbase.bucketcache.percentage.in.combinedcache
11
Bucket Cache Configura<on
Item id Values
Total RS memory Tot
Memstore size MSz
L1 (LRU) Cache L1Sz
Heap for JVM JHSz
XX:MaxDirectMemorySize DMem Tot-‐MSz-‐L1Sz-‐JHSz
Xmx Xmx MSz+L1Sz+JHSz
hbase.regionserver.global.memstore.upperLimit ULim MSz/Xmx
hfile.block.cache.size blksz 0.8-‐ULim
hbase.bucketcache.size bucsz Dmem+(blksz*Xmx)
hbase.bucketcache.percentage.in.combinedcache ccsz 1-‐((blksz*Xmx)/bucsz))
hbase.bucketcache.ioengine Ooeap/”file:/localfile”
12
Bucket Cache Configura<on
Item id Values
Total RS memory Tot 96000
Memstore size MSz 2000
L1 (LRU) Cache L1Sz 2000
Heap for JVM JHSz 1000
XX:MaxDirectMemorySize DMem 91000
Xmx Xmx 5000
hbase.regionserver.global.memstore.upperLimit ULim 0.4
hfile.block.cache.size blksz 0.4
hbase.bucketcache.size bucsz 93000
hbase.bucketcache.percentage.in.combinedcache ccsz 0.97849
hbase.bucketcache.ioengine ”file:/localfile”
13
Impact of Table Blk Size Change
Avg 3.002 5.362 5.361 5.357 6.419 6.369 6.405 6.383 6.188 6.196 6.182 6.174 6.246 6.264 6.268 6.253 5.194 5.207 5.219 3.031Median 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 295% 10 15 15 15 18 18 18 18 18 18 17 17 18 18 18 18 15 15 15 1099% 15 26 26 26 30 30 30 30 28 28 28 28 29 29 29 29 25 24 25 15
99.90% 26 41 41 41 45 45 45 45 43 43 43 43 44 44 44 44 41 41 41 26Max 2261 127 185 102 90 106 92 102 93 106 119 114 89 140 132 82 81 150 93 1910
BAvg 16.731 16.728 16.761 16.763 16.418 16.371 16.37 16.431 16.152 16.14 16.169 16.158 16.308 16.29 16.325 16.307 16.34 16.381 16.391 16.352BMedian 14 14 14 14 13 13 13 13 15 15 15 15 13 13 13 13 13 13 13 13B95% 41 41 41 41 41 41 41 41 43 43 43 43 40 40 40 40 41 41 41 41B99% 55 55 55 55 54 54 54 54 55 55 55 55 54 54 54 54 54 54 55 54B99.9% 71 71 71 71 70 70 70 70 67 67 67 67 71 70 70 71 71 71 71 70BMax 545 1062 559 567 1075 1027 561 567 564 541 558 1062 1062 561 1075 1072 1067 563 1035 1032
Get Performance (ms) – 64 K Blk
Get Performance (ms) – 16 K Blk
Note: Smaller block size increases the overhead of increased index blocks 14
Impact of Using Off-‐Heap Cache Get Performance with L1 cache
Get Performance with L1 & L2 cache
Note: L1 cache test used 38 GB og data, L1+L2 test used 3 TB of data
Avg 3.872 3.995 3.936 4.007 4.052Median 1 1 1 1 195% 14 14 14 15 1599% 20 20 20 20 20
99.90% 27 27 27 28 2899.99% 36 36 36 37 3799.999% 208 310 332 207 232Max 1360 1906 1736 1359 1363
807Mil797107Mil7Requests
BAvg 3.429 2.552 3.447 3.502 3.554BMedian 2 2 2 2 2B95% 10 8 10 10 10B99% 18 14 18 18 18B99.9% 30 23 30 30 31BMax 78 1135 58 77 67
18Mil8Rows8>818Mil8Requests
15
Impact of Running Mul<ple RS/Node
• Throughput increased 50% when RS increased to 2 – Through put reduced on AWS cluster – There was no degrada<on on the response <me – Through put increase tapered aqer 3 RS per node
• Due to maintenance over head mul<-‐RS op<on was discounted
16
Known Issues
• Using “ooeap” op<on of BucketCache prevents RS start – [HBASE-10643] – Can be mitigated using tempfs
• LoadIncrementalHFiles doesn’t work with BucketCache – [HBase-10500].
• BucketCache for different block sizes is not configurable
17
Further Reading
• hXp://blog.asquareb.com/blog/2014/11/21/leverage-‐hbase-‐cache-‐and-‐improve-‐read-‐performance
• hXp://blog.asquareb.com/blog/2014/11/24/how-‐to-‐leverage-‐large-‐physical-‐memory-‐to-‐improve-‐hbase-‐read-‐performance
• hXps://issues.apache.org/jira/browse/HBASE-‐7404
• hXp://www.n10k.com/blog/blockcache-‐101/ • hXp://www.n10k.com/blog/blockcache-‐showdown/
18
