massively parallel postgres kubernetes operator for...gemfire spark object storage hdfs json, apache...

© Copyright 2018 Pivotal Software, Inc. All rights Reserved.

Kubernetes Operator for Massively Parallel Postgres

Goutam Tadi (@goutamtadi)Senior Software Engineer, Pivotal Software IncEmail: gp-kubernetes@pivotal.io

PGConf India 2019

Agenda

● Intro to Greenplum

● Kubernetes 101

● Greenplum for Kubernetes

○ Components

■ Greenplum Operator

■ Greenplum Cluster

● Demo

Massively Parallel Postgres

Greenplum

Greenplum Data Platform

ANALYTICALAPPLICATIONS

NATIVE INTERFACES

PIVOTALGREENPLUMPLATFORM

MULTI-STRUCTURED DATA

SOURCES &PIPELINES

Structured Data

JDBC, ODBC

SQL

ANSI SQL

FLEXIBLE DEPLOYMENT

LocalStorage

OtherRDBMSes

SparkGemFireCloudObject

StorageHDFS

JSON, Apache AVRO, Apache Parquet and XML

Teradata SQL

Other DB SQL

Apache MADlib

ML/Statistics/Graph

Python. R,Java, Perl, C

Programmatic

Apache SOLR

Text

PostGIS

GeoSpatial

Custom Apps BI / Reporting Machine Learning AI

On-Premises

NEXT GENERATION

DATA PLATFORM

KafkaETLSpringCloud

Data Flow

MassivelyParallel(MPP)

PostgreSQLKernel

PetabyteScale

Loading

QueryOptimizer(GPORCA)

WorkloadManager

PolymorphicStorage

Command Center

SQL Compatibility

(Hyper-Q)

DS AnalystsIT Dev

PublicClouds

PrivateClouds

FullyManaged

Clouds

5

Container Orchestration System

Kubernetes

Greenplum on Kubernetes 101

Kubernetes Master

Greenplum on Kubernetes 101

Kubernetes Master

kubelet kube-proxy docker

Node

kubelet kube-proxy docker

Node

Greenplum on Kubernetes 101

Kubernetes Master

Pod

kubelet kube-proxy docker

Node

Pod

kubelet kube-proxy docker

Node

Greenplum on Kubernetes 101

Kubernetes Master

Pod

kubelet kube-proxy docker

Node

Pod

kubelet kube-proxy docker

Node

Storage volumes

Greenplum on Kubernetes 101

Kubernetes Master

Pod

Postgres Container

kubelet kube-proxy docker

Node

Pod

Postgres Container

kubelet kube-proxy docker

Node

Storage volumes

Greenplum on Kubernetes 101

Kubernetes MasterGreenplum Service

Pod

Postgres container

kubelet kube-proxy docker

Node

Pod

Postgres Container

kubelet kube-proxy docker

Node

Storage volumes

Greenplum on Kubernetes

Node

Pod

segment-b-0

kubelet kube-proxy docker

Pod

segment-a-0

kubelet kube-proxy docker

Node

Storage volumes

Pod

master-0

kubelet kube-proxy docker

Pod

master-1

kubelet kube-proxy docker

Massively Parallel Postgres on Kubernetes

Greenplum for Kubernetes

Components ● Greenplum Operator

● Greenplum Cluster

Greenplum Operator

apiVersion: "greenplum.pivotal.io/v1"kind: "GreenplumCluster"metadata: name: my-greenplumspec: masterAndStandby: …. Segments: ….

Greenplum Operator

Greenplum Cluster

CREATE / UPDATE / DELETE

Greenplum Cluster

Namespace: defaultKubernetes Cluster

Master StatefulSet

master-0 master-1

Greenplum Cluster

Namespace: defaultKubernetes Cluster

Primary StatefulSet

segment-a-0 segment-a-1

Mirror StatefulSet

segment-b-0 segment-b-1 Master StatefulSet

master-0 master-1

Greenplum Cluster

Namespace: defaultKubernetes Cluster

Primary StatefulSet

segment-a-0 segment-a-1

Mirror StatefulSet

segment-b-0 segment-b-1

ConfigMap

Master StatefulSet

master-0 master-1

Greenplum Cluster

Namespace: defaultKubernetes Cluster

Primary StatefulSet

segment-a-0 segment-a-1

Mirror StatefulSet

segment-b-0 segment-b-1

ConfigMap

Master StatefulSet

master-0 master-1

Greenplum Cluster

Namespace: defaultKubernetes Cluster

Primary StatefulSet

segment-a-0 segment-a-1

Mirror StatefulSet

segment-b-0 segment-b-1

ConfigMap

Master StatefulSet

master-0 master-1

Psql query

Benefits

● Declarative style deployments

● Auto cluster initialization

● Quick and fast deployments

● Easy to expand

● Delete the compute and retain storage for later use

Demohttps://youtu.be/d8X2BXSg07Q

Install Greenplum OperatorInstall Greenplum Cluster

Failover ScenariosExpand Greenplum Cluster

Future

Auto FailoverAuto Cluster Rejoin

PVC SnapshotPod Affinity

Transforming How The World Builds Software

© Copyright 2017 Pivotal Software, Inc. All rights Reserved.25

© Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0

Resources

http://greenplum-kubernetes.docs.pivotal.iohttps://network.pivotal.io