Application Renovation with Couchbase Clint Ricker

VSRM Intro

VSRM = Video Session & Resource Manager

Enables broadcast, on demand, and hosted DVR Video

Background scenario

Changing industry

Changing ecosystem

Sold as a 1U appliance

VSRM (Before Couchbase)

Scalability via segmentation (1 server per 100K subscribers)

No centralized management

VSR<

Platform Components

Applications

Application Protocol Adaptors

SDV Policy Mgr

S6VERM

SSP

Resource Manager

Resource Manager Adaptors

CMTS QAMBulk

Encryptor

Elem

ent Ma

nager (proprietary local

database)

VSRM (Before Couchbase)

MCMIS

VOD

Log Manager

Config Manager

HA Manager

VSRM

Clients (externally sharded to

~100K)

QAMQAMQAMQAMQAM

Mgmt

DB

VSRM

Clients (externally sharded to

~100K)

QAMQAMQAMQAMQAM

Mgmt

DB

VSRM

Clients (externally sharded to

~100K)

QAMQAMQAMQAMQAM

Mgmt

DB

VSRM

Clients (externally sharded to

~100K)

QAMQAMQAMQAMQAM

Mgmt

DB

VSRM

Clients (externally sharded to

~100K)

QAMQAMQAMQAMQAM

Mgmt

DB

VSRM Scalability (Before Couchbase)

Phase 1: Assessment

Strong Underlying Design

Pluggable components

Core components plus adaptors

Model based applications

Deep customer entrenchment

Large scale customer deployments

Deep customer integration and customization

Rich algorithms and business logic

Complex algorithms

Rich support for customer policy rules

Why not?

Bad Design

Easy to reimplement algorithms and business logic

Small customer entrenchment

High technical debt

Phase 2: Renovation

VSRM VSRM VSRM VSRM VSRM

DB DB DB DB DB

Concept

Distributed database / shared memory

Why Couchbase?

Strong consistency

Reliable low latency

Great scalability

VSR<

Platform Components

Applications

Application Protocol Adaptors

SDV Policy Mgr

S6VERM

SSP

Resource Manager

Resource Manager Adaptors

CMTS QAMBulk

Encryptor

Elem

ent Ma

nager (proprietary local

database)

VSRM

MCMIS

VOD

Log Manager

Config Manager

HA Manager

Couchbase Adaptor

DB Manager

State

Data Former Became

Per session data (session state)

In memory C struct and legacy local database

Couchbase keys (now text based)

Component data model (config / application state)

Legacy local database 1 couchbase key per tableSession data typically peeled offCounters are individual keys

Raw memory C struct Couchbase keys using memcpy on C/C++.Other APIs transparently handle objects.

Data Model Transformation

Data Sizing

Smaller keys have lower latency

Max rate of consistent update of a key–higher for small keys

Transactions needed for consistent updates of groups of keys

Design Rules

Real time operations use key/value operations

Views reserved for maintenance operations

All database calls go through abstraction layer

Locking / Serialization

Pessimistic locks as a replacement for mutex

Provides serialization, not order

Optimistic locks for read efficiency

Queries

Be intentional on what data is indexed

Weaker consistency

Limited resource, especially if stale=true

Data Model TransformationCouchbase = clustering Swiss Army knife

Lightweight directory services

Simple distributed queues with append

Master election via get w/lock

Danger Zones

Consider key contention. Break apart keys if necessary. Redesign to avoid.

Distributed databases require robust error handling.

Test error handling and edge cases at scale.

Phase 3: Makeover Grand Reveal

VSR<

Platform Components

Applications

Application Protocol Adaptors

SDV Policy Mgr

S6VERM

SSP

Resource Manager

Resource Manager Adaptors

CMTS QAMBulk

Encryptor

Elem

ent Ma

nager (proprietary local

database)

VSRM (Before Couchbase)

MCMIS

VOD

Log Manager

Config Manager

HA Manager

VSR<

Platform Components

Couchbase Adaptor

Applications

Application Protocol Adaptors

SDV Policy Mgr

S6VERM

SSP

Resource Manager

Resource Manager Adaptors

CMTS QAMBulk

Encryptor

Elem

ent Ma

nager (structure only)

VSRM Today

MCMIS

VOD

Log Manager

Config Manager

HA Manager

Web Svc Manager

Recorder Mgr

Linear Mgr

C3 A8RM-IS

RM-SS

RecorderVideo Mux

DB Manager

VSRM

Clients (externally sharded to

~100K)

QAMQAMQAMQAMQAM

Mgmt

DB

VSRM

Clients (externally sharded to

~100K)

QAMQAMQAMQAMQAM

Mgmt

DB

VSRM

Clients (externally sharded to

~100K)

QAMQAMQAMQAMQAM

Mgmt

DB

VSRM

Clients (externally sharded to

~100K)

QAMQAMQAMQAMQAM

Mgmt

DB

VSRM

Clients (externally sharded to

~100K)

QAMQAMQAMQAMQAM

Mgmt

DB

VSRM Scalability Before Couchbase

VSR<VSRMVM

Instance

VSRM Scalability Today (unified deployment model)

QAMQAMQAMQAMQAM

QAMQAMQAMQAMQAM

QAMQAMQAMQAMQAMQAMQAMQAMQAMQAM

QAMQAMQAMQAMQAM

QAMQAMQAMQAMQAM

QAMQAMQAMQAMQAM

QAMQAMQAMQAMQAM

QAMQAMQAMQAMQAM

Clients

VSRMVM

Instance

VSRMVM

Instance

VSRMVM

Instance

VSRMVM

Instance

Management

Load Balancer

Load Balancer

CouchbaseClusterCouchbase

ClusterCouchbaseCluster

Provisioning

Public or P

rivate C

loud

VSR<

VSRM Scalability Today (per service deployment model)

QAMQAMQAMQAMQAM

QAMQAMQAMQAMQAM

QAMQAMQAMQAMQAMQAMQAMQAMQAMQAM

QAMQAMQAMQAMQAM

QAMQAMQAMQAMQAM

QAMQAMQAMQAMQAM

QAMQAMQAMQAMQAM

QAMQAMQAMQAMQAM

Clients

App Micro Service

App Service

Load Balancer

Load Balancer

CouchbaseClusterCouchbase

ClusterCouchbaseCluster

Provisioning

Public or P

rivate C

loud

Resource Mgr

Service

Resource Mgr

Service

Resource Mgr

Service

App AdaptorsServices

Resource Mgr

Service

Resource AdaptorServices

Management

The End