pivotal hawq internals

1Pivotal Confidential–Internal Use Only 1Pivotal Confidential–Internal Use Only

Pivotal HAWQ

A.Grishchenko

HadoopKitchen @ Mail.ru27 Sep 2014

2Pivotal Confidential–Internal Use Only

SQL-on-Hadoop Solutions

Hive

2008

Developed by Facebook– Hive is used for data analysis in their data warehouse– DWH size is ~300PB at the moment, ~600TB of data is loaded daily. Data

is compressed using ORCFiles, compression ratio is ~8x

HiveQL language is not compatible with ANSI SQL-92

Has many limitations on subqueries

Cost-based optimizer (Optiq) is only in technical preview now



Hive

2008

Developed by Cloudera– Open-source solution– Cloudera sells this solution to enterprise shops– Was in beta until the May’2013

Supports HiveQL, moving forward complete ANSI SQL-92 support

Written in C++, does not use Map-Reduce for running queries

Requires much memory, big tables join usually causes OOM error

Impala

10.2012



Hive

2008

Hortonworks initiative– Consists of a number of steps to make Hive run 100x faster

Tez – solution to make Hive queries be translated to Tez jobs, which are similar to Map-Reduce but may have arbitrary topology

Optiq – cost-based query optimizer for Hive (technical preview ATM)

ORCFile – columnar storage format with adaptive compression and inline indexes

Hive-5317 – ACID and Update/Delete support (release at ~ 11.2014)

Impala

10.2012

Stinger

02.2013



Hive

2008

Pivotal product– Greenplum MPP DBMS, ported to store data in HDFS– Written in C, query optimizer is rewritten for this solution (ORCA)

Supports ANSI SQL-92 and analytic extensions from SQL-2003

Supports complex queries with correlated subqueries, window functions and different joins

Data is put on disk only if the process does not have enough memory

Impala

10.2012

Stinger

02.2013

HAWQ

02.2013



Hive

2008

HP Vertica– Supports only MapR distribution as requires updatable storage– Supports ANSI SQL-92, SQL-2003– Supports UPDATE/DELETE– Officially announced as available in July’2014, no implementations yet

IBM BigSQL v3– IBM DB2 ported to store data in HDFS– Federated queries, good query optimizer, etc.

Both solutions are similar to Pivotal HAWQ in general idea

Impala

10.2012

Stinger

02.2013

HAWQ

02.2013

Vertica,BigSQL

2014


Pivotal HAWQ Components

Master

Segment 1

Segment 2

Segment K

Server 1

Standby Master

Server 2

Server 3

Segment K+1

Segment K+2

Segment 2*K

Server 4

Segment N

Server M

… … ……



HAWQ Master

HAWQ Segm.

Server 1

HAWQ SBMstr

Server 2

Server 5

…

NameNode

Server 3

SNameNode

Server 4

ZK QJMZK QJMZK QJM

Datanode

HAWQ Segm.

Server 6

Datanode

HAWQ Segm.

Server M

Datanode



HAWQ Master

Query Parser

Query Optimizer

Query Executor

Transaction Manager

Process Manager

Metadata Catalog

HAWQ Standby Master

Query Parser

Query Optimizer

Query Executor

Transaction Manager

Process Manager

Metadata Catalog

WALreplic.


Pivotal HAWQ Components Metadata is stored only on master-servers

Metadata is stored in modified Postgres instance, replicated to standby master with WAL

Metadata contains– Table information – schema, names, files– Statistics – number of unique values, value ranges, sample values,

etc.– Information about users, groups, priorities, etc.

Master server shutdown causes the switch to standby with the loss of running sessions



HAWQ Segment

Query Executor

libhdfs3

PXF

HDFS Datanode

Segment Data Directory

Local Filesystem (xfs)

Spill Data Directory


Pivotal HAWQ Components Both masters and segments are modified postgres

instances (to be clear, modified Greenplum instances)

Opening connection to the master server you fork postmaster process that starts to work with your session

Starting the query execution you connect to the segment instances and they also fork a process to execute query

Query execution plan is split into independent blocks (slices), each of them is executed as a separate OS process on the segment server, moving the data through UDP


Pivotal HAWQ Components Tables can be stored as:

– Row-oriented (quicklz, zlib compression)– Column-oriented (quicklz, zlib, rle compression)– Parquet tables

Each segment has separate directory on HDFS where it stores its data shard

Within columnar storage each column is represented as a separate file

Parquet allows to store the table by columns and does not load NameNode with many files / block location requests


HAWQ Master

Metadata

Transact. Mgr.

Parser Query Optimiz.

Process Mgr.

Query Executor

NameNode

Query Execution in Pivotal HAWQ

Local Spill Directory

HAWQ SegmentBackend

HDFS Datanode

Segment Directory


HAWQ SegmentBackend

HDFS Datanode

Segment Directory


HAWQ Master

Metadata

Transact. Mgr.


Process Mgr.

Query Executor

NameNode



HAWQ SegmentBackend

HDFS Datanode

Segment Directory


HAWQ SegmentBackend

HDFS Datanode

Segment Directory


HAWQ Master

Metadata

Transact. Mgr.


Process Mgr.

Query Executor

NameNode



HAWQ SegmentBackend

HDFS Datanode

Segment Directory

QE


HAWQ SegmentBackend

HDFS Datanode

Segment Directory

QE


HAWQ Master

Metadata

Transact. Mgr.


Process Mgr.

Query Executor

NameNode



HAWQ SegmentBackend

HDFS Datanode

Segment Directory

QE S1 S2 S3


HAWQ SegmentBackend

HDFS Datanode

Segment Directory

QE S1 S2 S3


HAWQ Master

Metadata

Transact. Mgr.


Process Mgr.

Query Executor

NameNode



HAWQ SegmentBackend

HDFS Datanode

Segment Directory


HAWQ SegmentBackend

HDFS Datanode

Segment Directory


PXF Framework

Gives you ability to read different data types from HDFS– Text files, both compressed and uncompressed– Seqence-files– AVRO-files

Able to read data from external data sources– HBase– Cassandra– Redis

Extensible API


NameNode

PXF FrameworkHAWQ Master

PXF Fragmenter

Process Mgr.


HAWQ SegmentQuery Executor

HDFS DatanodeSegment Directory

PXF Accessor

PXF Fragmenter




PXF Accessor

PXF Fragmenter


NameNode


PXF Fragmenter

Process Mgr.




PXF Accessor

PXF Fragmenter




PXF Accessor

PXF Fragmenter


Further Steps

Master server scaling – pool of master servers

New native data storage formats and new native compression algorithms

YARN as resource manager for HAWQ

Dynamic segment allocation / decommission

45Pivotal Confidential–Internal Use Only 45Pivotal Confidential–Internal Use Only

Questions?

BUILT FOR THE SPEED OF BUSINESS

pivotal hawq internals

Software