Download - NYC Meetup November 15, 2012
![Page 1: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/1.jpg)
The Elastically Scalable Database™
1
![Page 2: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/2.jpg)
2
20th Century Database
Powerful Query Language
Industry Standards
Data Guarantees
Employee Skills
Tools
Existing Data
44%
21%
19%
4%3%
9%
ORACLEIBMMicrosoftSybaseTeradataOthers
![Page 3: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/3.jpg)
3
21st Century Problem Commodity Datacenters ✗
Big Data ✗ Modern Workloads ✗ 24x7 Operation ✗
Geo-distribution ✗ Developer Empowerment ✗
Powerful Query Language
Industry Standards
Data Guarantees
Employee Skills
Tools
Existing Data
![Page 4: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/4.jpg)
4
Database Crisis
Source: Marc Bojoly
Wikipedia Flickr
Amazon
![Page 5: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/5.jpg)
5
Jim Starkey
‣ DEC RDB/ELN ‣ InterBase ‣ Firebird ‣ Falcon ‣ BLOBS ‣ MVCC
“Elastically Scalable Transactions represent the biggest breakthrough in database technology in 25 years”
![Page 6: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/6.jpg)
6
" “An emergent behavior can appear when a number of simple entities operate in an environment, forming more complex behaviors as a collective.”
" - Wikipedia
Emergent Database Architecture
![Page 7: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/7.jpg)
7
Second Machine Instant Performance Increase
" Second machine typically doubles TPS
" Second machine is added to live database while it is running at 1,000’s of TPS
" Performance increase is immediate
" BTW - you can take either machine away and the database keeps running without data loss
NuoDB Plus One
![Page 8: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/8.jpg)
8
Second & Third Machine Instant Performance Increase
" Third machine typically triples single machine TPS
" Third machine is added to live database while it is running at 1,000’s of TPS
" Performance increase is immediate
" BTW - you can take any machine away and the database keeps running without data loss
Adding a Third Machine
![Page 9: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/9.jpg)
9
Number of Nodes
TPS " Technical Details: ‣ 2-9 Tx engines ‣ 1 storage
manager ‣ Best sustained TPS
and # clients combination
‣ 50% updates
Nodes TPS
MySQL 1 3,000 NuoDB 1 4,500
NuoDB 9 27,000
NuoDB running on 9 nodes was approx. 9x faster than MySQL running on 1 node.
More Machines? Bring ‘em On
![Page 10: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/10.jpg)
10
Number of EC2 Nodes
TPS
‣ Nuodb scales linearly on EC2
‣ Per-node performance on m1.large nodes approx 50% of our commodity servers
‣ Just started on optimizing
‣ RDS runs on 1 node, and gets overloaded with 10+ connections
Or Scale-out on IAAS
![Page 11: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/11.jpg)
Trek Palmer
![Page 12: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/12.jpg)
Building An Elastically Scalable Database The Easy Way
![Page 13: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/13.jpg)
13
Tonight’s Agenda " A Bit About Me " Introduction to NuoDB " Architecture Overview " Some Unnatural Acts
" Quick Demo " Beer
![Page 14: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/14.jpg)
14
A Bit About Me
" A Refugee from Academia " -Researched Programming Languages " -Transactional Memory impl. and
semantics " Worked on distributed metadata
database for HDS HCP " -Clustered appliance
![Page 15: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/15.jpg)
15
What is NuoDB?
" Elastically scalable " Multi-tenant " Transactionally Consistent " Easy to Manage
![Page 16: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/16.jpg)
16
娜 graceful, like a cloud
![Page 17: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/17.jpg)
Architecture " Three tiers
" Each is independent " Single model for all
environments " Extensible at
various points
Management
TransacEon Handling
Storage
![Page 18: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/18.jpg)
18
Agents
" Management tier " Provision hosts for use " Expose XML messaging for
management " Make scripting and automation easy
![Page 19: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/19.jpg)
19
Brokers
" Agent with additional special knowledge
" At least one per domain " Redirects clients to TE
" -Clients need no knowledge of topology
" -Brokers are responsible for any load-balancing
![Page 20: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/20.jpg)
20
Transaction Engines " Peer-to-peer " In-memory " Multi-Version Concurrency Control " Asynchronous messaging (replication) " Atoms
![Page 21: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/21.jpg)
21
Storage Managers
" Persistence points for atoms " Key-value backing stores " -Local FS, S3, HDFS
" Independent archives
![Page 22: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/22.jpg)
22
And Now, Some Unnatural Acts
![Page 23: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/23.jpg)
23
Sharding, an Unnatural Act " The ideal DB application
DB
client
client
Scales up to the capacity of a single node
What if you need more read and/or write throughput?
![Page 24: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/24.jpg)
24
Sharding
" Shard the DB among several nodes
DB0 DB1
Client1 Client2
New Client Layer Now you need to implement consistency in your applicaEon
TransacEonal consistency is very very hard to get right
![Page 25: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/25.jpg)
25
Other Sharding Bugbears
" Global operations (searches, scans) " -Doing joins in the application " -Implementing Cursors " -Chunking and memory
management " And, of course, adding or removing
shards
![Page 26: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/26.jpg)
26
Scaling Shards
" A recipe for changing the number of shards
" 1) Ask boss for permission " 2) Provision hardware " 3) Rewrite the app over 6 months " 4) Hope / Pray
![Page 27: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/27.jpg)
27
NuoDB Solution
Host A
Broker
Host B
Agent
Host C
Agent
Domain
Management Client
![Page 28: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/28.jpg)
28
NuoDB on a single node
Host A
Broker
Txn Engine
Host B
Agent
Storage Manager
Host C
Agent
Domain
Management Client
Database A
Client
![Page 29: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/29.jpg)
29
NuoDB Scaling out
Host A
Broker
Txn Engine
Host B
Agent
Storage Manager
Host C
Agent
Domain
Management Client
Database A
Client Client Client
Txn Engine
Storage Manager
![Page 30: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/30.jpg)
30
NuoDB ‘Adding a Shard’
Host A
Broker
Txn Engine
Host B
Agent
Storage Manager
Host C
Agent
Domain
Management Client
Database A
Client Client Client
Txn Engine
Storage Manager
Txn Engine
Storage Manager
![Page 31: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/31.jpg)
31
NuoDB ‘Sharding’
" Literally as simple as just adding nodes
" No client code had to be harmed in the making of this distributed database
" -Brokers hide topology changes " -NuoDB is transactionally consistent
![Page 32: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/32.jpg)
32
Eventual Consistency
" Eventual consistency is latent inconsistency
" -Not transactionally consistent " -Application porting is non-trivial " -Performance/Correctness tradeoff
icky
![Page 33: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/33.jpg)
33
NuoDB Consistency
" NuoDB is transactionally consistent " All the time, everywhere " When a transaction is committed,
it’s guaranteed consistent " Tradeoff is between Performance and
Availability
![Page 34: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/34.jpg)
34
Multi-Tenancy
" Traditional databases monopolize a node
" NuoDB supports many databases in a single pool of machines (a domain)
" Each DB can be scaled as needed, independantly
![Page 35: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/35.jpg)
35
Multi-Tenancy Example
Host A
Broker
Txn Engine
Host B
Agent
Storage Manager
Host C
Agent
Domain
Management Client
Database A
JDBC Client
![Page 36: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/36.jpg)
36
Multi-Tenancy Example
Host A
Broker
Txn Engine
Host B
Agent
Storage Manager
Host C
Agent
Domain
Management Client
Database A
JDBC Client
Txn Engine
JDBC Client
![Page 37: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/37.jpg)
37
Multi-Tenancy Example
Host A
Broker
Txn Engine
Host B
Agent
Storage Manager
Host C
Agent
Domain
Management Client
Database A
JDBC Client
Txn Engine
JDBC Client
Storage Manager
Txn Engine
Database 1
![Page 38: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/38.jpg)
38
Multi-Tenancy Example
Host A
Broker
Txn Engine
Host B
Agent
Storage Manager
Host C
Agent
Domain
Management Client
Database A
JDBC Client
Txn Engine
JDBC Client
Storage Manager
Txn Engine
Database 1
SQL Client ...
![Page 39: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/39.jpg)
And now, a demo…
![Page 40: NYC Meetup November 15, 2012](https://reader034.vdocuments.site/reader034/viewer/2022042814/554f4f7fb4c905524c8b4d4f/html5/thumbnails/40.jpg)
The Elastically Scalable Database™
40