greenplum on pks - 数字化三部曲 · kubelet kube-proxy docker pod kubernetes 101 . greenplum...

© Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0

翁岩⻘青 [email protected]

Greenplum on PKS Containerized MPP Database

1

Cover w/ Image

Agenda

■  Greenplum Architecture

■  Greenplum Data Platform

■  Kubernetes on PCF

■  Greenplum on Kubernetes

■  Q+A

2

OPEN SOURCE MASSIVELY PARALLEL PROCESSING

DATA WAREHOUSE

WHAT IS GREENPLUM?

3

Greenplum = Massively Parallel Postgres for Analytics

Standby Master

…

Master Host

SQL

Interconnect

Segment Host

Node1

Segment Host

Node2

Segment Host

Node3

Segment Host

NodeN

Local Storage

Other RDBMSes Spark GemFire

Cloud Object

Storage HDFS Kafka ETL

Spring Cloud

Data Flow

Master Servers Query planning and dispatch

Segment Servers Query processing and data storage

Interconnect

External Sources & Pipelines Parallel loading and streaming

Greenplum Data Platform

ANALYTICAL APPLICATIONS

NATIVE INTERFACES

PIVOTAL GREENPLUM PLATFORM

MULTI- STRUCTURED DATA

SOURCES & PIPELINES

Structured Data

JDBC, ODBC

SQL

ANSI SQL

FLEXIBLE DEPLOYMENT

Local Storage

Other RDBMSes Spark GemFire

Cloud Object

Storage HDFS

JSON, Apache AVRO, Apache Parquet and XML

Teradata SQL

Other DB SQL

Apache MADlib

ML/Statistics/Graph

Python. R, Java, Perl, C

Programmatic

Apache SOLR

Text

PostGIS

GeoSpatial

Custom Apps BI / Reporting Machine Learning AI

On-Premises

NEXT GENERATION

DATA PLATFORM

Kafka ETL Spring Cloud

Data Flow

Massively Parallel (MPP)

PostgreSQL Kernel

Petabyte Scale

Loading

Query Optimizer

(GPORCA)

Workload Manager

Polymorphic Storage

Command Center

SQL Compatibility

(Hyper-Q)

DS Analysts IT Dev

Public Clouds

Private Clouds

Fully Managed Clouds

5

Faster Deployments… How?

6

Have you Have you... ●  Ran out of disk space ?

●  Been able to provision more than 100 postgres instances in few minutes?

●  Faced issues in recovering failures ?

●  Faced issues in expanding the database?

7

Experienced these before with any database?

RELIABLY DEPLOY AND RUN CONTAINERIZED WORKLOADS.

WHAT IS PKS?

8

Kubernetes on Pivotal Cloud Foundry

Continuously deliver any app to every major private and public cloud with a single platform.

9

Cloud Native Apps & Greenplum

BOSH

Other Broker

Services

Platform Services

Logging Metrics Monitoring

Pivotal Application Service

VMware GCP Azure Openstack AWS

Cloud Native App

PK

S C

ontro

ller GCP

Service Broker

Harbor

NS

X-T

Kubernetes

Greenplum ML, GeoSpatial, Graph,

GPText, PXF PL/Python, PL/R

SQL

11

Faster Deployments… How?

Greenplum Data Platform + PKS

12

Kubernetes 101

Kubernetes Master

13

Kubernetes Master

kubelet kube-proxy docker

Node


Node

14

Kubernetes 101

Kubernetes Master

Pod


Node

Pod


Node

15

Kubernetes 101

Kubernetes Master

Pod


Node

Pod


Node

16

Kubernetes 101

Kubectl

Kubernetes Master

Pod


Node

Pod


Node

Storage volumes

17

Kubernetes 101

Kubernetes Master

Pod


Node

Pod


Node

Storage volumes

18

Kubernetes 101

Kubernetes Master Load Balancer Service

Node Node

Storage volumes

19

Pod


Node

Pod

Node

Kubernetes 101

Greenplum on Kubernetes

Node

Pod

mirror


20

Greenplum Service

Pod


standby

Pod


Node

Storage volumes

primary

Pod


master

Deployment topology for Private Cloud Local Storage for Performance & Remote Storage for Flexibility

Benefits

Greenplum on PKS

22

1.  On Demand Cluster Provisioning

PKS

Radar

23


PKS

Radar

24


PKS

Radar

Cluster Radar

25


PKS

Radar

Cluster Radar

26


PKS

Radar

Cluster Radar psql gpdb-radar:5432

27


PKS

Radar


28

Amy


PKS

Radar


29

Amy


PKS

Radar


Cluster Amy

30

Amy


PKS

Radar


Cluster Amy

31

Amy


PKS

Radar

Cluster Radar

Cluster Amy

psql gpdb-radar:5432

psql gpdb-amy:5432

32

Amy

2. Service Discovery

master

Container Pod

We can always discover a container by DNS. For example, DNS address for different roles:

master.greenplum.svc.cluster.local standby.greenplum.svc.cluster.local segment-0a.greenplum.svc.cluster.local segment-0b.greenplum.svc.cluster.local

standby

Container Pod

Mirror segment-0b

Container Pod

Primary segment-0a

Container Pod

33

3. HA without Rebalancing

seg-0a

Container Pod

seg-0b

Container Pod

seg-1a

Container Pod

seg-1b

Container Pod

Primary

Mirror

34

3. HA without Rebalancing

seg-0a

Container Pod

seg-0b

Container Pod

seg-1a

Container Pod

seg-1b

Container Pod

Mirror becomes primary And no need for rebalancing.

Primary

Mirror

35

4. Kubernetes Plugins Support : Container Storage Interface

36

seg-1b

Container Pod

Cloud Storage

4. Kubernetes Plugins Support : Logging

37

seg-0b

Container Pod

seg-1b

Container Pod

syslog / stderr syslog / stderr

Logging Agent Log Store

HEY PKS! GIVE ME A GREENPLUM CLUSTER OF

“N” SEGMENTS

GREENPLUM ON PKS DEMO

38

Deploy Greenplum on PKS

Demo

39

Expand Greenplum on PKS

Demo

41

Greenplum Segment Failover

Demo

43

How to How to

●  Auto segment failover

●  Auto activate standby if master failed

●  Auto expand cluster

45

Automate Greenplum operation on PKS

Greenplum Operator on Kubernetes

Greenplum on PKS

46

Kubernetes Operator

Custom Resource Definitions (CRD) ●  Provides API object framework for a new

resource

●  Used to implement new resource types ○  Data model for Controllers

Controllers

●  Active reconciliation process ○  Current State -> Desired State

●  Used to automate app administration ○  {add, extend, replace} cluster

functionality

Configuration and creation of the greenplum-operator Greenplum Operator Config

greenplum-operator uses DBA’s config to deploy Greenplum cluster Greenplum Cluster Deployment by Operator

Then DBA runs gpinitsystem or that step can be automated.

Quick, Consistently Repeatable, Pre-Hardened Deployments Greenplum Operator

Future: More automation and self-healing

More Persistent Volumes Custom Scheduler

Resource Management

Future Work

51

Q & A QUESTIONS?

52

greenplum on pks - 数字化三部曲 · kubelet kube-proxy docker pod kubernetes 101 . greenplum...

Documents