Real Time Analytics with HBase

Ivo Mitov datafusion.eu2016

Introduction● Integration gateway● Authentication, authorization,

throttling, routing, transformation and orchestration

● Multiple partners ● Multiple backends● High-volume data - 10 000 TPS

Monitoring and analytics ● Nagios● Oracle● Hadoop cluster● MapReduce jobs● Lack of real time information:

partner invocations error ratelatency

HBaseBigTable - sparse, distributed,

persistent multidimensional, sorted map

Column-oriented DBMS on top of HDFSRow key, column family, column

qualifier, version, value

HBase

HBase CoprocessorsEndpointsObservers

RegionObserverRegionServerObserverWALObserverMasterObserver

HBase clientAttribute descriptorMetric definition

filter expressionChannel adapterNomenclature managerHBase gateway

HBase schema● NТAttribute (name,’a’:value,id)● NТMetric (name,’m’:[type]

[attributes],id)● TEvent ([TID][CID],’e’:

[metrics],timestamp)● TMetric

([MID][rounded TS],’m’:type,agg value)([MID][rounded TS], ’m’:[type][TID]

[CIS],value)

TEvent coprocessor

TMetric coprocessor

Thank you!

ivo.mitov@datafusion.eu