beyond 1001 dedicated data service instances - … service instances. introduction. the challenge....

Beyond 1001 Dedicated Data Service Instances

Introduction

The Challenge

Given: Application platform based on

Cloud Foundry to serve thousands of apps

Application Runtime

App App App App App App App App App App










• Many platform users - who don’t

know each other

• Different app langs & frameworks

• 100% on-demand self-service: no

involvement of the platform

operator necessary.

• Instant scalablity & self-healing

$> cf push myapp

Easy Deployment

Runtime abstraction

ExecutionStagingJava

Droplet

Java

Code

Java

Bildpack

Droplet

ExecutionStaging

Ruby

Bildpack

DropletRuby Ruby

Droplet

Droplet DropletJava

Droplet Ruby

Droplet

In front of the Cloud

Foundry Runtime

all droplets are equal

Droplet Droplet

$START_CMD $START_CMD

DropletContainer

Image


Droplet

$START_CMD

Something you can

execute in a container.

Something you can



Something you can


$START_CMD

Abstraction enables further

assumptions & automation

$>

Scaling Apps

Assuming this to be our status quo.

Cloud Foundry Runtime

App#1

Instance#1

App#2

Instance#1

App#2

Instance#2

$> cf scale -i 3 app#1

App Scalability

$>

Scaling Apps


App#1

Instance#1

App#1

Instance#3

App#2

Instance#1

App#2

Instance#2

App#1

Instance#2

Two additional instances have been created.

App Self-Healing

$>

App Self-Healing

Everything is healthy.


App#1

Instance#1

App#1

Instance#3

App#2

Instance#1

App#2

Instance#2

App#1

Instance#2

$>


App Self-Healing

App #1 Instance #2 is failing.

App#1

Instance#1

App1

Instance

2

App#1

Instance#3

App#2

Instance#1

App#2

Instance#2

$>


App Self-Healing

App #1 Instance #2 - gone temporarily.

App#1

Instance#1

App#1

Instance#3

App#2

Instance#1

App#2

Instance#2

$>

App Self-Healing

App #1 Instance #2 re-created.


App#1

Instance#1

App#1

Instance#3

App#2

Instance#1

App#2

Instance#2

App#1

Instance#2

How does the paradise for

Backing Services look

like?

Missing: A solution to serve thousands of

data services

Data ServicesApplication Runtime











The Mission

Providing a growing number of data services with full lifecycle automation of

thousands of data service instances across a wide range of infrastructures



a9s PostgreSQL a9s MongoDB a9s RabbitMQ

a9s Elasticsearch a9s Redis a9s LogMe


thousands of data service instances across a wide range of infrastructures.

in both:

in both:public

in both:public and on-premise clouds

and integrate well with multiple platforms

Requirements

Portability

Robustness

Scalability

On-demand self-service

Performance Maintainability

Flexibility

Extensibility

Usability

Manageability

Security

Multi-tenancy

Portability

Scalability


Production-Readiness

Design

How to build it?

Data Service Provisioning API Automation Middleware Data Service Automation

Open Service Broker, a new industry standard

for data service provisioning.

🔑

• Google

• Pivotal

• IBM

• RedHat

• Fujitsu

• SAP

Open Service Broker API Supporters

• Cloud Foundry

• OpenShift

• Kubernetes

• More to come

Supporting Platforms

• Get Service CatalogGET /v2/catalog

• Provision Service - Create Service InstancePUT /v2/service_instances/:id

• Bind ServicePUT /v2/service_instances/:instance_id/

service_bindings/:id

• Unbind ServiceDELETE /v2/service_instances/:instance_id/


• Unprovision ServiceDELETE /v2/service_instances/:id

http://docs.cloudfoundry.org/services/api.html#api-overview

http://docs.cloudfoundry.org/services/api.html#api-overview

HTTP Verb Action

Service Catalog GET /v2/catalog

Deliver meta data about the data service.

Create Service Instance PUT /v2/service_instances/:id

Provision a VM, install and configure a data service VMs

/ Cluster representing a service instance.

Create Service Binding PUT /v2/service_instances/:instance_id/


Create a data service user and return credentials representing a service

binding.

Delete Service Binding DELETE /v2/

service_instances/:instance_id/service_bindings/:id

Remove credentials associated with the service binding.

Delete Service Instance DELETE /v2/service_instances/:id

Destroy the VMs and data associated with the service instance.

Data Service Provisioning API Automation Middleware Data Service

The Open Service Broker API does not define what a service instance is.

🔑

Applying the design pattern: On-Demand Provisioning

of Dedicated Data Service Instances

🔑

Result

Using a Service Broker with Cloud Foundry

$> cf create-service

$> cf create-service mongodb single-small my-single-mongo-1

Easy Deployment

my-single-mongo-1

MongoDB

VM#1

$> cf create-service mongodb cluster-small my-3node-mongo-

cluster-2

Easy Deployment

Newly created service instance

my-3node-mongo-cluster-2

MongoDB

VM#1

MongoDB

VM#2

MongoDB

VM#3

my-single-mongo-1

MongoDB

VM#1

Technical Challenges

State is handled differently in each backing service.


Operational model will be different. Replication, failure detection, failover.

⇓


Operational model will be different. Replication, failure detection, failover.

⇓

The data service automation will be different.

⇓

Where to store state?

App Self-Healing

Everything is healthy.


App#1

Instance#1

App#1

Instance#3

App#2

Instance#1

App#2

Instance#2

App#1

Instance#2


App Self-Healing

App #1 Instance #2 is failing.

App#1

Instance#1

App1

Instance

2

App#1

Instance#3

App#2

Instance#1

App#2

Instance#2


App Self-Healing

App #1 Instance #2 - gone temporarily.

App#1

Instance#1

App#1

Instance#3

App#2

Instance#1

App#2

Instance#2

App Self-Healing

App #1 Instance #2 re-created.


App#1

Instance#1

App#1

Instance#3

App#2

Instance#1

App#2

Instance#2

App#1

Instance#2

App self-healing is easy because there is NO STATE.

App self-healing is easy because there is NO STATE.

How to store state but still being able to perform self-healing?

⇓

Store state on a remotely attached block device = persistent disk.

🔑

Infrastructure as a Service (IaaS), e.g. OpenStack

VIRTUAL DATACENTER

Router

STORAGE

Storage Node Storage Node Storage Node

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

IaaS API


VIRTUAL DATACENTER

Router

STORAGE


HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

Storage Volume

Operating System

VIRTUAL MACHINE

IaaS API

Persistent disk has a file system. Filesystems may fail

⇒ replication / clustering & backupsare still very important.

The data lifecycle has been decoupled from the VM lifecycle ⇒ The VM becomes disposable.

🔑

Storing state ✅

What needs to be automated?

Data Service Instance Lifecycle

Provision a data service server Install data service software

Configure data service software Consume data service with apps

Debug data service issues Update data service version

Update operating system Backup & recover data

Scale out data service VM(s) Destroy data service & DB VM(s)

Lifecycle of a Data Service Instance1

Can you do that x * 1000 times?

You either automate it or delegate it to the app developer.

🔑

Automation

„BOSH let’s you orchestrate the

lifecycle of

large-scale deployments of stateful distributed systems

to infrastructure.“


VIRTUAL DATACENTER

VIRTUAL MACHINE

BOSH

Router

STORAGE


HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

BOSH API

BOSH CLI $> bosh deploy

IaaS API

BOSH CPI


VIRTUAL DATACENTER

VIRTUAL MACHINE

BOSH

Router

STORAGE


HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

BOSH API

Storage Volume

Operating System

BOSH Agent

VIRTUAL MACHINE


IaaS API

BOSH CPI


VIRTUAL DATACENTER

VIRTUAL MACHINE

BOSH

Router

STORAGE


HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

BOSH APIOperating

System

BOSH Agent

VIRTUAL MACHINE


IaaS API

BOSH CPI


VIRTUAL DATACENTER

VIRTUAL MACHINE

BOSH

Router

STORAGE


HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

BOSH APIOperating

System

BOSH Agent

VIRTUAL MACHINE


IaaS API

PostgreSQL

BOSH CPI


VIRTUAL DATACENTER

VIRTUAL MACHINE

BOSH

Router

STORAGE


HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

HDD HDD

BOSH APIOperating

System

BOSH Agent

VIRTUAL MACHINE


IaaS API

…

BOSH Agent

VIRTUAL MACHINE

PostgreSQL Operating System

BOSH Agent

VIRTUAL MACHINE

Cloud Controller

Operating System

BOSH Agent

VIRTUAL MACHINE

UAA

BOSH CPI

BOSH Automation

BOSH Releases

contain the automation

A BOSH-Deployment depends on 1 .. *

Stemcells

A BOSH-Deployment is described by a

Release & Manifest

A Release describe 1 .. * Jobs

A Release contains 1 .. * Package

A BOSH Deployment’s

settings are contained in a

Manifest

Infrastructure settings

settings are contained in the

Cloud Config

BOSH makes your

deployments…

Infrastructure

Independent

• A BOSH release contains the

main-automation (software packages, how to run

processes)

• BOSH releases can be re-used

on every* infrastructure

Automate once,

deploy everywhere.

VMware

BOSH

BOSH CLI

OpenStack

BOSH

AWS

BOSH

VMware

BOSH

BOSH CLI

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

OpenStack

BOSH

AWS

BOSH

BOSH CLI

VMware AWS OpenStack

BOSH BOSH BOSH

BOSH CLI


$> bosh target http://bosh-on.aws.com

BOSH BOSH BOSH

http://bosh-on.aws.com

BOSH CLI


BOSH BOSH BOSH

BOSH CLI


$> bosh deploy

BOSH BOSH BOSH

BOSH CLI


Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

BOSH BOSH BOSH

• Switch deployment between

clouds

• Keep the same release

• Use a stemcell specific to the

new cloud

• Adapt the cloud config

Operating System

Independent

A BOSH release does

not depend on the OS

The only dependency to

the OS is a BOSH

stemcell

Operating System Image

BOSH Agent

VIRTUAL MACHINE


BOSH Agent

VIRTUAL MACHINE

}


BOSH Agent

VIRTUAL MACHINE

} OS image +BOSH agent= Stemcell

Ubuntu Stemcell

BOSH Agent

VIRTUAL MACHINE

• Changing the OS of a BOSH

deployed system


• Change the stemcell

• Change the manifest

Scalable

Horizontal Scaling

Horizontal Scaling

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

• Scaling-out a BOSH deployed

system


• Use the same stemcell

• Change the manifest

Vertical Scaling

4GB RAM 1 vCPU

10GB persistent disk

BOSH Agent

VIRTUAL MACHINE 4 GB RAM, 1 vCPU

10 GB Persistent Disk

Data

PostgreSQL

BOSH Agent


PostgreSQL


Data

BOSH Agent


PostgreSQL

BOSH Agent

PostgreSQL

VIRTUAL MACHINE 8 GB RAM, 2 vCPUs


Data


Data

BOSH Agent

PostgreSQL

VIRTUAL MACHINE 8 GB RAM, 2 vCPUs


Data

BOSH Deployments are Predictable

• Src code is compiled in a freshly

created VMs

• VMs always contain exact the

software, specified in the release

• No left-overs of prior

deployments as new VMs are

used.

BOSH Deployments are

Repeatable

Executing a specific BOSH deployment always leads to exact same deployed system.

🔑

Monitored &Self-Healing

Self-healingprocess failures

BOSH Installation BOSH Managed Infrastructure Resources

Some process

BOSH Agent

VIR

TU

AL

MA

CH

INE

Process Monitor

Yet another process

VIR

TU

AL

MA

CH

INE

BOSH Agent

BOSH Health Monitor

BOSH Director

NATS Message Bus

Some other process

VIR

TU

AL

MA

CH

INE

BOSH Agent Process Monitor

BOSH Registry

BLOB Store

❤

🎖

🗃

🗄

📡

CP

I

Process MonitorProcess Monitor


Some process

BOSH Agent

VIR

TU

AL

MA

CH

INE

Process Monitor

Yet another process

VIR

TU

AL

MA

CH

INE

BOSH Agent

BOSH Health Monitor

BOSH Director

NATS Message Bus

VIR

TU

AL

MA

CH

INE


BOSH Registry

BLOB Store

❤

🎖

🗃

🗄

📡

CP

I

Some other process


Self-healingprocess monitor failures


Some process

BOSH Agent

VIR

TU

AL

MA

CH

INE

Process Monitor

Yet another process

VIR

TU

AL

MA

CH

INE

BOSH Agent

BOSH Health Monitor

BOSH Director

NATS Message Bus

Some other process

VIR

TU

AL

MA

CH

INE


BOSH Registry

BLOB Store

❤

🎖

🗃

🗄

📡

CP

I

Some other process



Some process

BOSH Agent

VIR

TU

AL

MA

CH

INE

Yet another process

VIR

TU

AL

MA

CH

INE

BOSH Agent

BOSH Health Monitor

BOSH Director

NATS Message Bus

Some other process

VIR

TU

AL

MA

CH

INE


BOSH Registry

BLOB Store

❤

🎖

🗃

🗄

📡

CP

I

Some other process



Some process

BOSH Agent

VIR

TU

AL

MA

CH

INE

Yet another process

VIR

TU

AL

MA

CH

INE

BOSH Agent

BOSH Health Monitor

BOSH Director

NATS Message Bus

Some other process

VIR

TU

AL

MA

CH

INE


BOSH Registry

BLOB Store

❤

🎖

🗃

🗄

📡

CP

I

Some other process


Process Monitor

Self-healingVM failures


Some process

BOSH Agent

VIR

TU

AL

MA

CH

INE

Process Monitor

Yet another process

VIR

TU

AL

MA

CH

INE


NATS Message Bus

Some other process

VIR

TU

AL

MA

CH

INE


Process Monitor

BOSH Registry

BLOB Store

🗃

🗄

📡

🎖 BOSH Director

CP

I

BOSH Health Monitor❤


Some process

BOSH Agent

VIR

TU

AL

MA

CH

INE

Process Monitor

Yet another process

VIR

TU

AL

MA

CH

INE


NATS Message Bus

Process Monitor

BOSH Registry

BLOB Store

🗃

🗄

📡

🎖 BOSH Director

CP

I

Some other process

VIR

TU

AL

MA

CH

INE



Self-healingBOSH Agent failures

BOSH Managed Infrastructure Resources

Yet another process

VIR

TU

AL

MA

CH

INE

Process Monitor

BOSH Installation

Some process

BOSH Agent

VIR

TU

AL

MA

CH

INE

Process Monitor

BOSH Agent

NATS Message Bus

Some other process

VIR

TU

AL

MA

CH

INE


BOSH Registry

BLOB Store

🗃

🗄

📡

🎖 BOSH Director

CP

I

BOSH Agent


BOSH Managed Infrastructure Resources

BOSH Installation

Some process

BOSH Agent

VIR

TU

AL

MA

CH

INE

Process Monitor

NATS Message Bus

Some other process

VIR

TU

AL

MA

CH

INE


BOSH Registry

BLOB Store

🗃

🗄

📡

🎖 BOSH Director

CP

I

BOSH Agent


Yet another process

VIR

TU

AL

MA

CH

INE

Process MonitorBOSH Agent

Reference Architecture

a9s Deployer

Templates Deployments

Bosh

a9s Service Broker

my-3node-postgres-cluster-2

Postgresql

VM#1

Postgresql

VM#2

Postgresql

VM#3

my-single-postgres-1

Postgresql

VM#1

Middleware Adapter

Cloud Foundry Adapter

a9s PostgreSQL SPI

Service InstanceService Instance


Postgresql

VM#1

Postgresql

VM#2

Postgresql

VM#3

Service Instance

…

Cloud ControllerCF Clientcreate service

create service

create deployment from template xy with attributes {…}

deploy release abc & deployment manifest xyz

Execute deployments

create

service specific credentials

create binding

PortabilityBOSH:

Multi-infrastructure support

Open Service Broker API:

Multi-platform support

Scalability On-demand provisioning of dedicated service instances

BOSH: Scale existing service instances

vertically, solo & clustered

instances

Production Readyness

Dedicated data service instances / Strong instance

isolation

BOSH: Self-healing, clustered service

instances, backup & restore


Open Service Broker API, On-demand provisioning, on-

demand updatesOn-demand backup &

restore

a9s PostgreSQL a9s MongoDB a9s RabbitMQ

a9s Elasticsearch a9s Redis a9s LogMe

Operations

Continuous Data Service Delivery

Testa9s

Release

Delivering Data Service Patches

BuildUpstream

Release

Platform #1

Platform #2

Platform #n

Updating the Data Services

Update Data Service Instances

a9s PostgreSQLOpen Source

PostgreSQL Building new Data Service Releases

Common Maintenance Tasks to performed at Scale

• Create VM

• Install and start data services

Create Service Instance

• Destroy old VM

• Create new VM

• Mount old persistent disk

• Create and mount new persistent disk

• Copy data

• Optional: reintegrate into the cluster

Vertical Scale Service Instance

• Destroy old VM

• Create new VM based on new

Stemcell (\w new OS version)

• Attach persistent disk

OS Update

Ultimate Question

Can you handle more than 1001 Data Service Instances?

Excerpt from our perf tests:

Provisioning 1001 instances in sequence over a greater period of time does not

expose any significant bottleneck.

For large highly-frequented platforms the amount of simultaneous deployments

may become relevant.

Data Service Instances

BOSH Queue Time

Avg. time to provision VM

Total time needed to

create instances

250 14 min 6:57 min 21 min

500 29 min 7:01 36 min

750 46 min 7:13 53 min

… … … …

Optimization Task: Manage BOSH queueing time to an acceptable

level.

Scaling BOSH is key to deal with simultaneous provisioning.

Sum Up

• Full lifecycle automation is feasible

• Open Service Broker API, a new

standard

• Choosing the right automation

technology is key, e.g. BOSH

• CI/CD based dev and ops are essential

Sum Up

Questions?

A. Shared VM cluster

B. Dedicated containers

C. Dedicated VMs / VM clusters

Common Data Service Design Patterns

Scaling a shared VM cluster

3 VMs

MongoDB Cluster

MongoDB VM #1 MongoDB VM #2 MongoDB VM #3


3 VMs

MongoDB Cluster


Service Instance #1 = database #1






Service Instance #… = database #…

Service Instance # n-max = database #n-max


Low costs per service instance


Simple Service Broker Logic


create service → create a database

create service binding → create a database user

Weak Isolation!

Structural Limitation!

What to do when the shared cluster is full?












3 VMs

MongoDB Cluster









3 VMs


MongoDB Cluster #1










3 VMs

MongoDB Cluster









3 VMs


MongoDB Cluster #1










3 VMs


MongoDB Cluster #1










3 VMs


MongoDB Cluster #2

Service Instance #n+1






Service Instance #…

Service Instance # 2*n-max

Service Broker Logic


Simple



Complex

Fragmentation

Caused by frequent creation and / deletion of

service instances

Fragmentation

3 VMs

MongoDB VM #2 MongoDB VM #3

MongoDB Cluster #1









3 VMs


MongoDB Cluster #2










service instances

Fragmentation

3 VMs


MongoDB Cluster #1






3 VMs


MongoDB Cluster #2





Placement Problem

Placement Problem

3 VMs


MongoDB Cluster #1






3 VMs


MongoDB Cluster #2





Placement Problem

3 VMs


MongoDB Cluster #1






3 VMs


MongoDB Cluster #2





New Service Instance?

Strategy to place new service instances is required

and may require data service specific logic.

Placement Problem

3 VMs


MongoDB Cluster #1






3 VMs


MongoDB Cluster #2





New Service Instance

Cluster Rebalancing

Cluster Rebalancing

3 VMs


MongoDB Cluster #1






3 VMs


MongoDB Cluster #2





A unbalanced set of clusters wastes

infrastructure resources.

Cluster Rebalancing

3 VMs


MongoDB Cluster #1






3 VMs


MongoDB Cluster #2



Cluster Rebalancing

3 VMs


MongoDB Cluster #1






3 VMs


MongoDB Cluster #2



A cluster rebalance freeing infrastructure

resources would desirable.

Cluster Rebalancing

3 VMs


MongoDB Cluster #1








Shared Cluster Conclusion

• Scalability issued can be addressed

• Isolation issues are heavily data-service

specific > A generic solution is not

possible.

Scaling Dedicated Containers

Better Isolation

2 VMs across 2 AZs

PosgreSQL Cell

Docker host VM #1 Docker host VM #2


2 VMs across 2 AZs

PosgreSQL Cell



Service Instance #1 = 2 Docker containers

+ 2 PostgreSQL processes

+ 2 PostgreSQL databases asynchronously replicated

2 VMs across 2 AZs

PosgreSQL Cell









How to scale?

2 VMs across 2 AZs

PosgreSQL Cell









What to do when the Cell/Cluster is full?



2 VMs across 2 AZs

PosgreSQL Cell #1






2 VMs across 2 AZs

PosgreSQL Cell #2






Same Service Broker Challenge

New Challenge: How to add Cell-VMs on-demand?

On-Demand VM provisioning is unavoidable.

Why not delegate most challenges?

On-Demand Dedicated VMs and Clusters

Architecture

a9s Deployer


Bosh

a9s Service Broker

my-3node-mongodb-cluster-2

MongoDB

VM#1

MongoDB

VM#2

MongoDB

VM#3

my-single-mongodb-1

MongoDB

VM#1

Middleware Adapter


a9s MongoDB SPI



MongoDB

VM#1

MongoDB

VM#2

MongoDB

VM#3

Service Instance

…


create service



Execute deployments

create


create binding

Let BOSH do the VM orchestration!

Let the infrastructure solve the placement and fragmentation challenge!

Shared Data Services

• Single VM or single

cluster of VMs

• Single PostgreSQL

server or single

PostgreSQL cluster

• Isolation limited to

PostreSQL multi-

tenancy capabilities

Shared PostgreSQL Cluster > Bad idea

3 VMs

Postgresql

VM#1

Postgresql

VM#2

Postgresql

VM#3

1 VM

Postgresql

VM#1

PostgreSQL Cluster

Single PostgreSQL Server

Service Instance

Service Instance 2

Service Instance 3

Service Instance 1

Service Instance 2

Service Instance 3

Service Instance 1

Service Instance 2

Service Instance 3

Service Instance 1

Service Instance 2

Service Instance 3

Service Instance 1

OR

1x

1x

Shared PostgreSQL = SPOF


3 VMs

PostgreSQL Cluster

Postgresql

VM#1

Postgresql

VM#2

Postgresql

VM#3Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance












3 VMs

PostgreSQL Cluster

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

App App App App

App App App App

App App App App App

App App App App App App

App App

App App

App App

App App App

App

App App App App App App App



App App App App App

App App App App

App App App App App App App App




App App App App App App App App App

App App App

Service Instance

Service Instance

Service Instance


3 VMs

PostgreSQL Cluster

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

App App App App

App App App App

App App App App App


App App

App App

App App

App App App

App


Service Instance

Service Instance

Service Instance

Your shared PostgreSQL cluster goes down, all your PostgreSQL database instances go down.

Beware of bad neighborhood


3 VMs

PostgreSQL Cluster

Postgresql

VM#1

Postgresql

VM#2

Postgresql


Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance












3 VMs

PostgreSQL Cluster

Postgresql

VM#1

Postgresql

VM#2

Postgresql


Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance











👹

👹 👹

👹

👹 👹

👹 👹


3 VMs

PostgreSQL Cluster

Postgresql

VM#1

Postgresql

VM#2

Postgresql


Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance











Service Instance ⚠

👹

👹 👹

👹

👹 👹

👹 👹


3 VMs

PostgreSQL Cluster

Postgresql

VM#1

Postgresql

VM#2

Postgresql


Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance













Service Instance ⚠ Service Instance ⚠





👹

👹 👹

👹

👹 👹

👹 👹

Shared clusters are vulnerable to bad neighbors

Dedicated

• Service instance =

dedicated VM or

dedicated cluster of

VMs

• Uses infrastructure

isolation to enable

multi-tenancy support

Dedicated PostgreSQL instances > Good idea


Postgresql

VM#1

Postgresql

VM#2

Postgresql

VM#3


Postgresql

VM#1

Service Instance

Service Instance

n x

m x

and / or












Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql

VM#1

Postgresql

VM#2

Postgresql

VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

PostgreSQL failures are contained. Only one service instance affected.

Bad neighborhood protection with dedicated service instances


3 VMs

PostgreSQL Cluster = Service Instance #2

Postgresql

VM#1

Postgresql

VM#2

Postgresql

VM#3Service Instance Service Instance Service Instance











3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql


3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql



3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql












👹

👹 👹

👹

👹 👹

👹 👹

3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql


3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql



3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql












Service Instance ⚠ Service Instance ⚠ Service Instance ⚠

👹

👹 👹

👹

👹 👹

👹 👹

3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql


3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql



3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql













👹

👹 👹

👹

👹 👹

👹 👹

3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql


3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql


✅

✅


3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql













👹

👹 👹

👹

👹 👹

👹 👹

3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql


3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql


✅

✅

Infrastructure isolation

Dedicated clusters isolate bad neighbors

Cloud

Automation

Cloud

Automation

Robustness

Cloud

Automation

Self-Healing

Robustness

Cloud

Automation

Self-Healing

Scalability

Robustness

Cloud

Automation

Self-Healing

Scalability


Robustness

VM vs. Container

Resource TypePre-provisioned

vs. On-demand-provisioning

Provisioning

Single replica vs.

Data redundancy / Multiple replicas

Resurrection Failover vs.

Standby-Failover

Failover Strategy

Data Redundancy

Automation Technology

BOSH vs. Chef vs. Puppet

Service InstancesShared Dedicated

Infrastructure Reliability

Perfect. HA VMs. No SPOFs. Never fails. VMs cost more than a

Design to fail. Fails from time to time.

Saves money.

Seconds, minutes, hours?

Desired time to repair Service instance availability

Service broker availability

Availability

Service broker performance (ops/s) Service instance performance

Time to provision service instance.

PerformanceAccessibilityRemote log-in to service instances.

ConfigurabilityAdapt to local network

and security policies. Integrate existing

infrastructure.

SecurityNetwork security.

Encryption.

TransparencyAccessing metrics and logs.

OperabilityEasyness to operate and

maintain.

1. Define a service instance! 2. Define total # service instances! 3. Define # service instance CRUD ops / min!

Common Data Service

Design Patterns

A. Shared VM cluster

B. Dedicated containers

C. Dedicated VMs / VM clusters

Common Data Service Design Patterns

3 VMs

MongoDB Cluster



Low costs per service instance


Simple Service Broker Logic


create service → create a database

create service binding → create a database user

Weak Isolation!

What to do when the shared cluster is full?












3 VMs

MongoDB Cluster









3 VMs


MongoDB Cluster #1










3 VMs

MongoDB Cluster









3 VMs


MongoDB Cluster #1










3 VMs


MongoDB Cluster #1










3 VMs


MongoDB Cluster #2











Simple



Complex

Fragmentation


service instances

Fragmentation

3 VMs


MongoDB Cluster #1









3 VMs


MongoDB Cluster #2










service instances

Fragmentation

3 VMs


MongoDB Cluster #1






3 VMs


MongoDB Cluster #2





Placement Problem

Placement Problem

3 VMs


MongoDB Cluster #1






3 VMs


MongoDB Cluster #2





Placement Problem

3 VMs


MongoDB Cluster #1






3 VMs


MongoDB Cluster #2





New Service Instance?

Strategy to place new service instances is required

and may require data service specific logic.

Placement Problem

3 VMs


MongoDB Cluster #1






3 VMs


MongoDB Cluster #2





New Service Instance

Cluster Rebalancing

Cluster Rebalancing

3 VMs


MongoDB Cluster #1






3 VMs


MongoDB Cluster #2





A unbalanced set of clusters wastes

infrastructure resources.

Cluster Rebalancing

3 VMs


MongoDB Cluster #1






3 VMs


MongoDB Cluster #2



Cluster Rebalancing

3 VMs


MongoDB Cluster #1






3 VMs


MongoDB Cluster #2



A cluster rebalance freeing infrastructure

resources would desirable.

Cluster Rebalancing

3 VMs


MongoDB Cluster #1








Shared Cluster Conclusion

• Scalability issued can be addressed

• Isolation issues are heavily data-service

specific > A generic solution is not

possible.

Better Isolation

2 VMs across 2 AZs

PosgreSQL Cell



How to scale?

2 VMs across 2 AZs

PosgreSQL Cell









What to do when the Cell/Cluster is full?



2 VMs across 2 AZs

PosgreSQL Cell #1






2 VMs across 2 AZs

PosgreSQL Cell #2






Same Service Broker Challenge

New Challenge: How to add Cell-VMs on-demand?

On-Demand VM provisioning is unavoidable.

Why not delegate most challenges?

On-Demand Dedicated VMs and Clusters

Architecture

a9s Deployer


Bosh

a9s Service Broker


MongoDB

VM#1

MongoDB

VM#2

MongoDB

VM#3

my-single-mongodb-1

MongoDB

VM#1

Middleware Adapter


a9s MongoDB SPI



MongoDB

VM#1

MongoDB

VM#2

MongoDB

VM#3

Service Instance

…


create service



Execute deployments

create


create binding

Let BOSH do the VM orchestration!

Let the infrastructure solve the placement and fragmentation challenge!

Shared Services Instances

• Single VM or single

cluster of VMs

• Single PostgreSQL

server or single

PostgreSQL cluster

• Isolation limited to

PostreSQL multi-

tenancy capabilities

Shared PostgreSQL Cluster > Bad idea

3 VMs

Postgresql

VM#1

Postgresql

VM#2

Postgresql

VM#3

1 VM

Postgresql

VM#1

PostgreSQL Cluster

Single PostgreSQL Server

Service Instance

Service Instance 2

Service Instance 3

Service Instance 1

Service Instance 2

Service Instance 3

Service Instance 1

Service Instance 2

Service Instance 3

Service Instance 1

Service Instance 2

Service Instance 3

Service Instance 1

OR

1x

1x

Shared PostgreSQL = SPOF


3 VMs

PostgreSQL Cluster

Postgresql

VM#1

Postgresql

VM#2

Postgresql


Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance












3 VMs

PostgreSQL Cluster

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

App App App App

App App App App

App App App App App


App App

App App

App App

App App App

App




App App App App App

App App App App





App App App App App App App App App

App App App

Service Instance

Service Instance

Service Instance


3 VMs

PostgreSQL Cluster

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

App App App App

App App App App

App App App App App


App App

App App

App App

App App App

App


Service Instance

Service Instance

Service Instance

Your shared PostgreSQL cluster goes down, all your PostgreSQL database instances go down.

Beware of bad neighborhood


3 VMs

PostgreSQL Cluster

Postgresql

VM#1

Postgresql

VM#2

Postgresql


Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance












3 VMs

PostgreSQL Cluster

Postgresql

VM#1

Postgresql

VM#2

Postgresql


Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance











👹

👹 👹

👹

👹 👹

👹 👹


3 VMs

PostgreSQL Cluster

Postgresql

VM#1

Postgresql

VM#2

Postgresql


Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance












👹

👹 👹

👹

👹 👹

👹 👹


3 VMs

PostgreSQL Cluster

Postgresql

VM#1

Postgresql

VM#2

Postgresql


Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance


















👹

👹 👹

👹

👹 👹

👹 👹

Shared clusters are vulnerable to bad neighbors

Dedicated Service Instances

• Service instance =

dedicated VM or

dedicated cluster of

VMs

• Uses infrastructure

isolation to enable

multi-tenancy support

Dedicated PostgreSQL instances > Good idea


Postgresql

VM#1

Postgresql

VM#2

Postgresql

VM#3


Postgresql

VM#1

Service Instance

Service Instance

n x

m x

and / or












Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql

VM#1

Postgresql

VM#2

Postgresql

VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

PostgreSQL failures are contained. Only one service instance affected.

Bad neighborhood protection with dedicated service instances


3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql












3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql


3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql



3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql












👹

👹 👹

👹

👹 👹

👹 👹

3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql


3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql



3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql













👹

👹 👹

👹

👹 👹

👹 👹

3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql


3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql



3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql













👹

👹 👹

👹

👹 👹

👹 👹

3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql


3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql


✅

✅


3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql













👹

👹 👹

👹

👹 👹

👹 👹

3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql


3 VMs


Postgresql

VM#1

Postgresql

VM#2

Postgresql


✅

✅

Infrastructure isolation

Dedicated clusters isolate bad neighbors

Pre- vs. On-Demand Provisioning

Pre-Provisioned

Pre-provisoned service instances


Postgresql

VM#1

Postgresql

VM#2

Postgresql

VM#3


Postgresql

VM#1


Service Broker

$> cf create-service postgresql single-small my-single-postgres-1

Easy Deployment

Pre-provisioned service instances


Postgresql

VM#1

Postgresql

VM#2

Postgresql

VM#3


Postgresql

VM#1


Service Broker

$> cf create-service postgresql cluster-small my-3node-postgres-

cluster-2

Easy Deployment

Pre-provisioned service instances


Postgresql

VM#1

Postgresql

VM#2

Postgresql

VM#3


Postgresql

VM#1


Service Broker

On-Demand-Provisioned

On-demand provisioned service instances

Service Broker

Some Automation

On-demand provisioned service instances


Postgresql

VM#1

Service Instance

Service Broker

Some Automation

Pre-provisoned service instances


Postgresql

VM#1

Postgresql

VM#2

Postgresql

VM#3


Postgresql

VM#1


Service Broker

Some Automation

beyond 1001 dedicated data service instances - … service instances. introduction. the challenge....

Documents