go simple-fast-elastic-with-couchbase-server-borkar

1

Go Simple, Fast, Elas1c with Couchbase Server and

Document-‐oriented Data Management

Dip1 Borkar Director, Product Management

2

WHY TRANSITION TO NOSQL?

3

Changes in interac1ve soDware – NoSQL driver

4

11%

12%

16%

29%

35%

49%

Other

All of these

Costs

High latency/low performance

Inability to scale out data

Lack of flexibility/rigid schemas

Source: Couchbase NoSQL Survey, December 2011, n=1351

What is the biggest data management problem driving your use of NoSQL in the coming year?

Survey: Two big drivers for NoSQL adop1on

5

COUCHBASE SERVER

6

Couchbase Server: Simple. Fast. Elas1c. NoSQL.

•  Easy scalability •  Scale your the database layer without any interruption or change to your application

•  Grow your cluster from 5 to 25 to 100 nodes and evenly distributed workloads across your cluster

•  Consistent High Performance •  Consistent sub-‐millisecond response times •  High throughput means fewer servers required to serve growing numbers of users

•  Always on •  With high availability features applica1ons are always

online •  Whether upgrading the database, system soDware or

hardware -‐ zero down1me

7

Flexible Data Model

•  Not need to worry about the database when changing your applica1on

•  Records can have different structures, there is no fixed schema •  Record structure can simply be changed at any 1me, without affec1ng remaining stored data

•  The JSON based document model also maps extremely well to how applica1ons represent data, including nested sturctures

{ “ID”: 1, “FIRST”: “DipR”, “LAST”: “Borkar”, “ZIP”: “94040”, “CITY”: “MV”, “STATE”: “CA” }

JSON JSON

JSON JSON

8

Rela1onal vs Document data model

RelaRonal data model Document data model Collec1on of complex documents with arbitrary, nested data formats and

varying “record” format.

Highly-‐structured table organiza1on with rigidly-‐defined data formats and

record structure.

JSON JSON

JSON

C1 C2 C3 C4

{ }

9

Example: User Profile

Address Info

1 DEN 30303 CO

2 MV 94040 CA

3 CHI 60609 IL

User Info

KEY First ZIP_id Last

4 NY 10010 NY

1 DipR 2 Borkar

2 Joe

2 Smith

3 Baxter 2 Dodson

4 Lari 3 Gorin

ZIP_id CITY ZIP STATE

1 2

2 MV 94040 CA

To get informaRon about specific user, you perform a join across two tables

10

Represen1ng rows as documents

Geo Info

1 DEN 30303 CO

2 MV 94040 CA

3 CHI 60609 IL

User Info

KEY First ZIP_id Last

4 NY 10010 NY

1 DipR 2 Borkar

2 Joe

2 Smith

3 Baxter 2 Dodson

4 Lari 3 Gorin

ZIP_id CITY ZIP STATE

= { “ID”: 1, “FIRST”: “DipR”, “LAST”: “Borkar”, “ZIP”: “94040”, “CITY”: “MV”, “STATE”: “CA” } JSON

All data in a single document

11

Couchbase Server 2.0 – Coming soon

•  Next major release of Couchbase Server

What’s new: •  Indexing and Querying •  Incremental Map Reduce •  Cross Data Center Replica1on

12

Paid Produc1on Deployments (par1al list)

13

COUCHBASE SERVER ARCHITECTURE

14

Couchbase Server 2.0 Architecture

Heartbeat

Process m

onito

r

Glob

al singleton supe

rviso

r

Confi

gura1o

n manager

on each node

Rebalance orchestrator

Nod

e he

alth m

onito

r

one per cluster

vBucket state and

replica1

on m

anager

hdp RE

ST m

anagem

ent A

PI/W

eb UI

HTTP 8091

Erlang port mapper 4369

Distributed Erlang 21100 -‐ 21199

Erlang/OTP

storage interface

Couchbase EP Engine

11210 Memcapable 2.0

Moxi


Memcached

New Persistence Layer

8092 Query API

Que

ry Engine

Data Manager Cluster Manager

15

Couchbase Server 2.0 Architecture

Heartbeat

Process m

onito

r

Glob

al singleton supe

rviso

r

Confi

gura1o

n manager

on each node

Rebalance orchestrator

Nod

e he

alth m

onito

r

one per cluster

vBucket state and

replica1

on m

anager

hdp RE

ST m

anagem

ent A

PI/W

eb UI

HTTP 8091

Erlang port mapper 4369

Distributed Erlang 21100 -‐ 21199

Erlang/OTP

storage interface

Couchbase EP Engine


Moxi


Memcached

New Persistence Layer

8092 Query API

Que

ry Engine

16

Couchbase deployment

Data Flow

Cluster Management

Web Applica1on

Couchbase Client Library

17

Couchbase Server Admin Console

18

3 3 2

Clustering With Couchbase

SET request arrives at KEY’s master server

Listener-‐Sender

Master server for KEY Replica Server 2 for KEY Replica Server 1 for KEY

1 1 SET acknowledgement returned to applica1on

2

Disk Disk Disk

RAM

Couchb

ase storage en

gine

Disk Disk Disk

4

19

COUCHBASE CLIENT LIBRARY

Basic Opera1on

§ Docs distributed evenly across servers in the cluster

§ Each server stores both ac2ve & replica docs §  Only one server ac1ve at a 1me

§ Client library provides app with simple interface to database

§ Cluster map provides map to which server doc is on §  App never needs to know

§  App reads, writes, updates docs

§  Mul1ple App Servers can access same document at same 1me

Doc 4

Doc 2

Doc 5

SERVER 1

Doc 6

Doc 4

SERVER 2

Doc 7

Doc 1

SERVER 3

Doc 3

User Configured Replica Count = 1

Read/Write/Update


Read/Write/Update

Doc 9

Doc 7

Doc 8 Doc 6

Doc 3

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

Doc 9

Doc 5

DOC

DOC

DOC

Doc 1

Doc 8 Doc 2

Replica Docs Replica Docs Replica Docs

Ac1ve Docs Ac1ve Docs Ac1ve Docs

CLUSTER MAP

CLUSTER MAP

APP SERVER 1 APP SERVER 2

COUCHBASE SERVER CLUSTER

20

Add Nodes

§  Two servers added to cluster §  One-‐click opera1on

§  Docs automa1cally rebalanced across cluster §  Even distribu1on of

docs §  Minimum doc movement

§  Cluster map updated

§  App database calls now distributed over larger # of servers


Read/Write/Update Read/Write/Update

Doc 7

Doc 9

Doc 3

Ac1ve Docs

Replica Docs

Doc 6

COUCHBASE CLIENT LIBRARY CLUSTER MAP

APP SERVER 1


APP SERVER 2

Doc 4

Doc 2

Doc 5

SERVER 1

Doc 6

Doc 4

SERVER 2

Doc 7

Doc 1

SERVER 3

Doc 3

Doc 9

Doc 7

Doc 8 Doc 6

Doc 3

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

Doc 9

Doc 5

DOC

DOC

DOC

Doc 1

Doc 8 Doc 2



SERVER 4 SERVER 5

Ac1ve Docs Ac1ve Docs

Replica Docs Replica Docs


21

Fail Over Node

§  App servers happily accessing docs on Server 3

§  Server fails §  App server requests to server 3 fail §  Cluster detects server has failed

§  Promotes replicas of docs to ac2ve §  Updates cluster map

§  App server requests for docs now go to appropriate server

§  Typically rebalance would follow


Doc 7

Doc 9

Doc 3

Ac1ve Docs

Replica Docs

Doc 6


APP SERVER 1


APP SERVER 2

Doc 4

Doc 2

Doc 5

SERVER 1

Doc 6

Doc 4

SERVER 2

Doc 7

Doc 1

SERVER 3

Doc 3

Doc 9

Doc 7 Doc 8

Doc 6

Doc 3

DOC

DOC

DOC DOC

DOC

DOC

DOC DOC

DOC

DOC

DOC DOC

DOC

DOC

DOC

Doc 9

Doc 5 DOC

DOC

DOC

Doc 1

Doc 8

Doc 2



SERVER 4 SERVER 5

Ac1ve Docs Ac1ve Docs

Replica Docs Replica Docs


22

Reading and Wri1ng

Reading Data WriRng Data

Server

Give me document A

Here is document A

Application Server

A

Server

Please store document A

OK, I stored document A

Application Server

A

RAM

DISK

A

A

RAM

DISK

A

A

23

Server

Flow of data when wri1ng

WriRng Data

Application ServerApplication Server Application Server

ApplicaRons wriRng to Couchbase

Couchbase wriRng to disk

network

Couchbase transmiing replicas

ReplicaRon queue Disk write queue

24

Indexing and Querying

APP SERVER 1


§ Indexing work is distributed amongst nodes §  Large data set possible §  Parallelize the effort

§ Each node has index for data stored on it

§ Queries combine the results from required nodes

CLUSTER MAP

Doc 4

Doc 2

Doc 5

SERVER 1

Doc 6

Doc 4

SERVER 2

Doc 7

Doc 1

SERVER 3

Doc 3

APP SERVER 2


Doc 9

Doc 7

Doc 8 Doc 6

Doc 3

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

Doc 9

Doc 5

DOC

DOC

DOC

Doc 1

Doc 8 Doc 2





CLUSTER MAP

APP SERVER 1 APP SERVER 2

Query Response

25

Couchbase SDKs

Java SDK

.Net SDK

PHP SDK

Ruby SDK

…and many more

Java client API

User Code

Couchbase Server

CouchbaseClient cb = new CouchbaseClient(listURIs, "aBucket", "letmein"); // this is all the same as before cb.set("hello", 0, "world"); cb.get("hello");

htp://www.couchbase.com/develop

Couchbase Java Library (spymemcached)

26

DEMO (IF TIME PERMITS)

27

Coming September 21, 2012

SAN FRANCISCO

Featured speakers include:

Tell them DipR sent you and get 50% off. Use discount code DIPTI. hdp://www.couchbase.com/couchconf-‐san-‐francisco

28

THANK YOU

[email protected]

go simple-fast-elastic-with-couchbase-server-borkar

Technology