building a production grade postgresql cloud foundry service

Building a produc.on grade PostgreSQL Cloud Foundry Service

Produc.on grade?

Produc.on Ready Litmus Test: Run a public CF.

Design Decisions

Defining service instances.

PostgreSQL Server vs. Cluster

Replica.on

Sync vs. Async

Async replica.on is built-in since PostgreSQL 9

PostgreSQL Replica.on Limita.ons

Failure Detec.on

Automa.c Failover

repmgr Replica.on Manager for

PostgreSQL clusters

• Monitoring replica.on performance • Failure detec.on • Automated failover

• Leader elec.on • Master promo.on

Shared vs DedicatedPostgreSQL VMs

Shared

• Single VM or single cluster of VMs

• Single PostgreSQL server or single PostgreSQL cluster

• Isola8on limited to PostreSQL mul8-tenancy capabili8es

Shared PostgreSQL Cluster > Bad idea

3 VMs

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

1 VM

Postgresql VM#1

PostgreSQL Cluster

Single PostgreSQL Server

Service Instance

Service Instance 2

Service Instance 3

Service Instance 1

Service Instance 2

Service Instance 3

Service Instance 1

Service Instance 2

Service Instance 3

Service Instance 1

Service Instance 2

Service Instance 3

Service Instance 1

OR

1x

1x

Shared PostgreSQL = SPOF

Cloud Foundry Run.me

3 VMs

PostgreSQL Cluster

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

App App App App App App App App App App











3 VMs

PostgreSQL Cluster

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

App App App App

App App App App

App App App App App

App App App App App App

App App

App App

App App

App App App

App

App App App App App App App



App App App App App

App App App App

App App App App App App App App




App App App App App App App App App

App App App

Service Instance

Service Instance

Service Instance


3 VMs

PostgreSQL Cluster

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

App App App App

App App App App

App App App App App


App App

App App

App App

App App App

App


Service Instance

Service Instance

Service Instance

Your shared PostgreSQL cluster goes down, all your PostgreSQL database instances go down.

Dedicated

• Service instance = dedicated VM or dedicated cluster of VMs

• Uses infrastructure isola8on to enable mul8-tenancy support

Dedicated PostgreSQL instances > Good idea

my-3node-postgres-cluster-2

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

my-single-postgres-1

Postgresql VM#1

Service Instance

Service Instance

n x

m x

and / or












Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

PostgreSQL failures are contained. Only one service instance affected.

Pre- vs. On-Demand Provisioning

Pre-Provisioned

Pre-provisoned service instances


Postgresql VM#1

Postgresql VM#2

Postgresql VM#3


Postgresql VM#1

Service InstanceService Instance

Service Broker

$> cf create-service postgresql single-small my-single-postgres-1

Easy Deployment

Pre-provisioned service instances


Postgresql VM#1

Postgresql VM#2

Postgresql VM#3


Postgresql VM#1


Service Broker

$> cf create-service postgresql cluster-small my-3node-postgres-

cluster-2

Easy Deployment

Pre-provisioned service instances


Postgresql VM#1

Postgresql VM#2

Postgresql VM#3


Postgresql VM#1


Service Broker

On-Demand-Provisioned

On-demand provisioned service instances

Service Broker

Some Automa.on

On-demand provisioned service instances


Postgresql VM#1

Service Instance

Service Broker

Some Automa.on

Pre-provisoned service instances


Postgresql VM#1

Postgresql VM#2

Postgresql VM#3


Postgresql VM#1


Service Broker

Some Automa.on

Automa.on Technology

Bosh = ❤

Infrastructure independence

VM and persistent disk management

OS independence

Blueprint vs. construc.on Bosh Release vs. Bosh deployment

One Bosh release, several service plans. PostgreSQL single vm vs. PostgreSQL cluster

Monitoring & Self-Healing


Postgresql VM#1

Postgresql VM#2

Postgresql VM#3


Postgresql VM#1

Postgresql VM#3


Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Scalability


Postgresql VM#1


Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

$> cf service-update my-single-postgres-1 -p cluster-large

Scalability


Postgresql VM#1

Postgresql VM#2

Postgresql VM#3


Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Single node postgresql service instance > turned into a 3-node postgresql cluster.

$> cf service-update \ my-3node-postgres-cluster-1 \

-p cluster-large

Scalability


Postgresql VM#1

Postgresql VM#2

Postgresql VM#3


Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Small cluster turned into a large cluster.

Architectural Overview

a9s Deployer

Templates Deployments

Bosh

a9s Service Broker


Postgresql VM#1

Postgresql VM#2

Postgresql VM#3


Postgresql VM#1

Middleware Adapter

Cloud Foundry Adapter

a9s PostgreSQL SPI



Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

…

Cloud ControllerCF Clientcreate service

create service

create deployment from template xy with a_ributes {…}

deploy release abc & deployment manifest xyz

Execute deployments

create service specific

creden.als

create binding

a9s Service Broker

• Implements the CF Service Broker API

• Generic • Data service specific

SPIs

a9s Service Broker

a9s PostgreSQL SPI

The PostgreSQL SPI encapsulates PostgreSQL specific logic

• Service catalog metadata, e.g. service plans

• PostgreSQL instance creden8al management (service bindings)

a9s PostgreSQL SPI

a9s Deployer

The a9s Deployer manages Bosh deployments

Component Interac.on

Lessons Learned

• Dedicated service instances are mandatory

• On-demand provisioning is essen.al • Bosh does a great job

The greatest challenge with PostgreSQL was

• Finding a PostgreSQL replica.on & clustering toolset that allows automa.on

• Itera.vely learn how to configure and automate common and edge-case scenarios.

Ques.ons?

@fischerjulian @anynines

Thank you.

@fischerjulian @anynines

building a production grade postgresql cloud foundry service

Software