softshake 2013 intro to nosql
TRANSCRIPT
Introduc)on to NoSQLwith Couchbase
Tugdual “Tug” Grall
Technical Evangelist
[email protected] TwiBer: tgrall
Monday, October 28, 13
• Tugdual “Tug” Grall
-‐ Couchbase
-‐ Technical Evangelist
-‐ eXo
-‐ CTO
-‐ Oracle
-‐ Developer/Product Manager
-‐ Mainly Java/SOA
-‐ Developer in consulAng firms
• Web
-‐ @tgrall
-‐ hEp://blog.grallandco.com
-‐ tgrall
• NantesJUG co-‐founder
• Pet Project :
• hEp://www.resultri.com
{“about” : “me”}
Monday, October 28, 13
RDBMS are not Enough?
Monday, October 28, 13
Growth is the New Reality
• Instagram gained nearly 1 million users overnight when then expanded to Android
Monday, October 28, 13
Draw Something by OMGPOPDaily Ac)ve Users (millions)
191715131197533/12826242220181614121082/6 21
2
4
6
8
10
12
14
16
50 Million Users in 50 Days
Monday, October 28, 13
RDBMS is good for many thing, but hard to scale
RDBMS Scales UpGet a bigger, more complex server
Users
Applica;on Scales OutJust add more commodity web servers
Users
System CostApplica;on Performance
Rela2onal Database
Web/App Server Tier
System CostApplica;on Performance
Won’t scale beyond this point
How do you take this growth?
Monday, October 28, 13
Web/App Server Tier
Memcached Tier
MySQL Tier
Scaling out RDBMS
• Run Many SQL Servers
• Data could be sharded Done by the applicaAon code
• Caching for faster response )me
Monday, October 28, 13
Scaling out fla?ens the cost and performance curves
NoSQL Database Scales OutCost and performance mirrors app ;er
Users
NoSQL Distributed Data Store
Web/App Server Tier
Applica;on Scales OutJust add more commodity web servers
Users
System CostApplica;on Performance
Applica;on Performance System Cost
NoSQL Technology Scales Out
Monday, October 28, 13
DynamoOctober 2007
CassandraAugust 2008
BigtableNovember 2006
VoldemortFebruary 2009
Very few organiza)ons want to (fewer can) build and maintain database so]ware technology.But every organiza)on building interac)ve web applica)ons needs this technology.
A New Technology?
Building new database to answer the following requirements
• No schema required before inser)ng data
• No schema change required to change data format
• Auto-‐sharding without applica)on par)cipa)on
• Distributed queries
• Integrated main memory caching
• Data synchroniza)on ( mul)-‐datacenter)
Monday, October 28, 13
Lack of flexibility/rigid schemas
Inability to scale out data
Performance challenges Cost All of these Other
49%
35%
29%
16% 12% 11%
Source: Couchbase Survey, December 2011, n = 1351.
What Is Biggest Data Management Problem Driving Use of NoSQL in Coming Year?
Monday, October 28, 13
Key-‐Value
Memcached
Membase
Redis
Data Structure Document Column Graph
MongoDB
Couchbase Cassandra
Cache
(mem
ory on
ly)
Database
(mem
ory/disk)
Neo4j
HBase InfiniteGraph
Coherence
NoSQL Catalog
Monday, October 28, 13
ClouderaHortonworks
Mapr
CouchbaseMongoDB
CassandraHbase
AnalyPcDatabases
Get insights from data
Real-‐Pme, InteracPve Databases
Fast access to data
NoSQL
Opera)onal vs. Analy)c Databases
Monday, October 28, 13
Document Database
Monday, October 28, 13
Rela;onal vs Document Data Model
RelaPonal data model Document data modelCollec;on of complex documents witharbitrary, nested data formats and
varying “record” format.
Highly-‐structured table organiza;on with rigidly-‐defined data formats and record
structure.
JSONJSON
C1 C2 C3 C4
JSON
{
}
Monday, October 28, 13
Rela;onal Approach
Monday, October 28, 13
Document Databaseo::1001{uid: “ji22jd”,customer: “Ann”,line_items: [ { sku: 0321293533, qty: 2, price: 48.0 },{ sku: 0321601912, qty: 1, price: 39.0 },{ sku: 0131495054, qty: 1, price: 51.0 } ],payment: { type: “Amex”, expiry: “04/2001”, last5: 12345 }}
Aggregate Oriented DatabasehEp://marAnfowler.com/bliki/AggregateOrientedDatabase.html
Monday, October 28, 13
Schema Update : RDBMS
Speaker
Robin Johnson
Tug Grall
John Zablo
Event InfoID Title Speaker_id
Frank Weigel
1 Paris JUG rbin
2 QCON UK tgrall
3 NoSQL CLN johnz
4 Devoxx fweigel
ID Name
rbin
tgrall
johnz
fweigel
A talk could be done by one or more speakers!
1. Create a new table
2.“Prepare” the Data (move to new table, drop column)
3. Change the constraints
4. Change the app
Monday, October 28, 13
Schema Update : Document
event:1{ type: “event”, title: “Paris JUG”, speaker: “rbin”}
event:1{ type: “event”, title: “Paris JUG”, speaker: “rbin”}
event:1{ type: “event”, title: “Paris JUG”, speaker: “rbin”}
event:1{ type: “event”, title: “Paris JUG”, speaker: “rbin”}
event:1{ type: “event”, title: “Paris JUG”, speaker: “rbin”}
event:1{ type: “event”, title: “Paris JUG”, speaker: [“rbin”,”tgrall”]}
user:rbin{ type: “user”, name: “Robin Johnson”,}
user:rbin{ type: “user”, name: “Robin Johnson”,}
user:rbin{ type: “user”, name: “Robin Johnson”,}
1. Change The ApplicaPon• Update the Data when “you” want• Change the index to query the new format
Monday, October 28, 13
Couchbase Server
Monday, October 28, 13
Easy Scalability
Consistent High Performance
Always On 24x365
Grow cluster without applicaAon changes, without downAme with
a single click
Consistent sub-‐millisecond read and write response Ames with consistent high throughput
No downAme for sodware upgrades, hardware maintenance,
etc.
Flexible Data Model
JSON document model with no fixed schema.
JSONJSONJSON
JSONJSON
PERFORMANCE
Couchbase Server Core Principles
Monday, October 28, 13
Monday, October 28, 13
Heartbeat
Process m
onito
r
Glob
al singleton supe
rviso
r
Confi
gura;o
n manager
on each node
Rebalance orchestrator
Nod
e he
alth m
onito
r
one per cluster
vBucket state and
replica;
on m
anager
hRpRE
ST m
anagem
ent A
PI/W
eb UI
HTTP8091
Erlang port mapper4369
Distributed Erlang21100 -‐ 21199
Erlang/OTP
storage interface
Couchbase EP Engine
11210Memcapable 2.0
11211Memcapable 1.0
Memcached
New Persistence Layer
8092Query API
Que
ry Engine
Data Manager Cluster Manager
Couchbase Architecture
Moxi
Monday, October 28, 13
New Persistence Layer
storage interface
Couchbase EP Engine
11210Memcapable 2.0
11211Memcapable 1.0
Object-‐level Cache
Disk Persistence
8092Query API
Que
ry Engine
HTTP8091
Erlang port mapper4369
Distributed Erlang21100 -‐ 21199
Heartbeat
Process m
onito
r
Glob
al singleton supe
rviso
r
Confi
gura;o
n manager
on each node
Rebalance orchestrator
Nod
e he
alth m
onito
r
one per cluster
vBucket state and
replica;
on m
anager
hRp
REST m
anagem
ent A
PI/W
eb UI
Erlang/OTP
Server/Cluster Management & Communica2on
(Erlang)
The Unreasonable Effectiveness of C by Damien Katz
Couchbase Architecture
Moxi
RAM Cache, Indexing & Persistence Management
(C & V8)
Monday, October 28, 13
hEps://github.com/couchbase/hEps://github.com/couchbaselabs/
hEp://review.couchbase.org/Gerrit:
Open Source ProjectApache 2.0
Monday, October 28, 13
Couchbase Server Cluster
Cluster-‐wide Basic Opera)on
• Docs distributed evenly across servers
• Each server stores both ac)ve and replica docsOnly one server acAve at a Ame
• Client library provides app with simple interface to database
• Cluster map provides map to which server doc is onApp never needs to know
• App reads, writes, updates docs
• Mul)ple app servers can access same document at same )me
User Configured Replica Count = 1
READ/WRITE/UPDATE
Ac2ve
Doc 5
Doc 2
Doc
Doc
Doc
Server 1
Ac2ve
Doc 4
Doc 7
Doc
Doc
Doc
Server 2
Doc 8
Ac2ve
Doc 1
Doc 2
Doc
Doc
Doc
REPLICA
Doc 4
Doc 1
Doc 8
Doc
Doc
Doc
REPLICA
Doc 6
Doc 3
Doc 2
Doc
Doc
Doc
REPLICA
Doc 7
Doc 9
Doc 5
Doc
Doc
Doc
Server 3
Doc 6
App Server 1
COUCHBASE Client LibraryCluster Map
COUCHBASE Client LibraryCluster Map
App Server 2
Doc 9
Monday, October 28, 13
Add Nodes to Cluster
• Two servers addedOne-‐click opera)on
• Docs automa)cally rebalanced across clusterEven distribuAon of docsMinimum doc movement
• Cluster map updated
• App database calls now distributed over larger number of servers
REPLICA
Ac2ve
Doc 5
Doc 2
Doc
Doc
Doc 4
Doc 1
Doc
Doc
Server 1
REPLICA
Ac2ve
Doc 4
Doc 7
Doc
Doc
Doc 6
Doc 3
Doc
Doc
Server 2
REPLICA
Ac2ve
Doc 1
Doc 2
Doc
Doc
Doc 7
Doc 9
Doc
Doc
Server 3 Server 4 Server 5
REPLICA
Ac2ve
REPLICA
Ac2ve
Doc
Doc 8 Doc
Doc 9 Doc
Doc 2 Doc
Doc 8 Doc
Doc 5 Doc
Doc 6
READ/WRITE/UPDATE READ/WRITE/UPDATE
App Server 1
COUCHBASE Client LibraryCluster Map
COUCHBASE Client LibraryCluster Map
App Server 2
Couchbase Server Cluster
User Configured Replica Count = 1
Monday, October 28, 13
Fail Over Node
REPLICA
Ac2ve
Doc 5
Doc 2
Doc
Doc
Doc 4
Doc 1
Doc
Doc
Server 1
REPLICA
Ac2ve
Doc 4
Doc 7
Doc
Doc
Doc 6
Doc 3
Doc
Doc
Server 2
REPLICA
Ac2ve
Doc 1
Doc 2
Doc
Doc
Doc 7
Doc 9
Doc
Doc
Server 3 Server 4 Server 5
REPLICA
Ac2ve
REPLICA
Ac2ve
Doc 9
Doc 8
Doc Doc 6 Doc
Doc
Doc 5 Doc
Doc 2
Doc 8 Doc
Doc
• App servers accessing docs
• Requests to Server 3 fail
• Cluster detects server failedPromotes replicas of docs to acAveUpdates cluster map
• Requests for docs now go to appropriate server
• Typically rebalance would follow
Doc
Doc 1 Doc 3
App Server 1
COUCHBASE Client LibraryCluster Map
COUCHBASE Client LibraryCluster Map
App Server 2
User Configured Replica Count = 1
Couchbase Server Cluster
Monday, October 28, 13
COUCHBASE SERVER CLUSTER
ACTIVE
Doc 5
Doc 2
Doc
Doc
Doc
SERVER 1
REPLICA
Doc 4
Doc 1
Doc 8
Doc
Doc
Doc
APP SERVER 1
COUCHBASE Client Library
CLUSTER MAP
COUCHBASE Client Library
CLUSTER MAP
APP SERVER 2
Doc 9
• Indexing work is distributed amongst nodes
• Large data set possible
• Parallelize the effort
• Each node has index for data stored on it
• Queries combine the results from required nodes
ACTIVE
Doc 5
Doc 2
Doc
Doc
Doc
SERVER 2
REPLICA
Doc 4
Doc 1
Doc 8
Doc
Doc
Doc
Doc 9
ACTIVE
Doc 5
Doc 2
Doc
Doc
Doc
SERVER 3
REPLICA
Doc 4
Doc 1
Doc 8
Doc
Doc
Doc
Doc 9
Query
Indexing and Querying
Monday, October 28, 13
www.couchbase.com/develop
Go Clojure
Community SDKs
PythonRuby
Official SDKs
Couchbase SDK
Monday, October 28, 13
• Elastic Search is good for ad-hoc queries and faceted browsing• Our adapter is aware of changing Couchbase topology• Indexed by Elastic Search after stored to disk in Couchbase
ElasPcSearch
Full Text Search
Monday, October 28, 13
The Complete Mobile Solu;on
Available now in Beta :hfp://mobile.couchbase.com
Monday, October 28, 13
Contact me on Twi?er@tgrall
Contact me by [email protected]
IRC#couchbase#libcouchbase
Couchbase Docswww.couchbase.com/docs/
Couchbase Forumswww.couchbase.com/communi;es
Meetuphbp://meetup.com/Couchbase-‐France/
Mobile (beta)hbp://mobile.couchbase.com/
N1QL -‐ New Query Engine (DP)hbp://query.couchbase.com
Monday, October 28, 13
Introduc)on to NoSQLwith Couchbase
Tugdual “Tug” Grall
Technical Evangelist
[email protected] TwiBer: tgrall
Monday, October 28, 13