Slide 1

HBase MTTR, Stripe Compaction Ted Yu (tyu@hortonworks.com)

Slide 2

HBase 3 2011 6 HBase PMC

Slide 3

Hbase Recovery Overview HDFS Stripe compaction

Slide 4

0.96 / 0.98 protobuf RPC (MTTR) (Namespace) Cell -KV (Visibility Tags)

Slide 5

0.96.1 0.96.0 Release3 YCSB clients (Puts / second) 5 YCSB clients (Puts / second) 0.94.14178K220K 0.96.020K15K 0.96.1173K168K Release10 YCSB clients (Reads / second) 25 YCSB clients (Reads / second) 0.94.1450K52K 0.96.023K 0.96.152K54K

Slide 6

, HBase .

Slide 7

HBase

Slide 8

Slide 9

: ZooKeeper . , : master : ZK Heartbeat Client Region Servers, DataNode Data recovery Master, RS, ZK Region Assignment

Slide 10

ZooKeeper 30 180 GC . ZooKeeper 0.96 (kill -9),HBase ZK => 0

Slide 11

: 180 30 : 10 : 10

Slide 12

DataNode ! WAL DataNode RegionServer 33 DataNode DN ( ) NameNode TBs , NameNode 10

Slide 13

HDFS Live Stale Dead : : . , HDFS TBs HBase 30 , . 10 ,

Slide 14

/ HDFS dfs.timeout 30 . , 30

Slide 15

Slide 16

. , . .

Slide 17

Master . 5 . . TCP . hbase.rpc.timeout

Slide 18

(HBASE-7006) HFiles Hfiles NameNode NameNode : ( ) : WAL

Slide 19

RegionServer0 RegionServer_x RegionServer_y WAL-file3 .. WAL-file2 .. WAL-file1 .. HDFS Splitlog-file-for-region3 .. Splitlog-file-for-region2 .. Splitlog-file-for-region1 .. HDFS RegionServer3 RegionServer2 RegionServer1 writes reads

Slide 20

RegionServer0 RegionServer_x RegionServer_y WAL-file3 .. WAL-file2 .. WAL-file1 .. HDFS Recovered-file-for-region3 .. Recovered-file-for-region2 .. Recovered-file-for-region1 .. HDFS RegionServer3 RegionServer2 RegionServer1 writes reads replays

Slide 21

memstore : : 30 , ! : HBASE-8701 Edits

Slide 22

Slide 23

MemStore flush : , memstore MEMSTORE MTTR

Slide 24

hbase:meta 0.96 hbase:meta , ( hbase:meta) hbase:meta MemStore flush

Slide 25

HBase 3 RegionServers RegionServer

Slide 26

HDFS-5016 , DataNode HBASE-9039 http://hortonworks.com/blog/hortonworks- scaled-risk-and-ebay-collaborate-to-improve- hbase-mean-time-to-recovery-mttr/ http://hortonworks.com/blog/hortonworks- scaled-risk-and-ebay-collaborate-to-improve- hbase-mean-time-to-recovery-mttr/

Slide 27

1 - /RegionServer

Slide 28

2 - /RegionServer

Slide 29

MVCC ThreadLocal ThreadLocal RegionScannerImpl readpoint Scanner classes ThreadLocal Readpoint

Slide 30

Compactions Architecting the Future of Big Data Memstore MemStore HDFS writes HFile

Slide 31

, compaction I/ O

Slide 32

Hortonworks Inc. 2011 compactions Compaction uncompacted OpenTSDB I/ O compactions I / O , : compactions HBase ; compactions

Slide 33

Hortonworks Inc. 2011 Stripe compactions (HBASE-7667) Architecting the Future of Big Data LevelDB, region/store , 1 ( L 0 ) regions, stripes , stripes L 0, HFile Region start key: ccceee Row-key axis iii: region end keyggg H H HFile L0 get 'hbase'

Slide 34

Hortonworks Inc. 2011 Stripe compactions Architecting the Future of Big Data MEMSTORE stripe Compact MemStore HDFS HFile H H H H H H H H

Slide 35

Hortonworks Inc. 2011 Stripe compactions Architecting the Future of Big Data Level 0 ? Bulk loaded L 0 L0 ( ) L0 striped stripe +L 0, deletes major compactions, 2 stripes , - stripes (Split)

Slide 36

Hortonworks Inc. 2011 Stripe compactions - Architecting the Future of Big Data EC2, c1.xlarge, preload; LoadTestTool + deletes + overwrites;

Slide 37

Q & A Thanks!