©2015 Couchbase Inc. 1
Migration and Coexistence between RDBMS and NoSQL databases
Manuel HurtadoSolutions Engineer
©2015 Couchbase Inc. 3
What is NoSQL?
©2015 Couchbase Inc. 4
DynamoOctober 2007
CassandraAugust 2008
BigtableNovember 2006
VoldemortFebruary 2009
Very few organizations want to (fewer can) build and maintain database software technology.But every organization participating in the digital economy needs this technology.
Why NoSQL
New data solutions to address these needs: No schema required before inserting data No schema change required to change data format Auto-Sharding without application participation Distributed queries Managed Cache Data Synchronization ( multi-datacenter)
©2015 Couchbase Inc. 5
What is NoSQL?
No SQL? No. Not only SQL? Not really. Non relational? Yes.
Distributed (most)– Scaled out, not up
• Elasticity and commodity hardware
– Partitioned and replicated• Scalability, performance, availability
Schema-less (most)– Flexible model– JSON (some)
©2015 Couchbase Inc. 6
World of Operational Databases
6
Graph Stores
Neo4j
RDBMS/Transactional Stores
Oracle, DB2, SQLServer, MySql, Postgres
Caching Stores
Coherence, Redis, Gemfire
Full-text Stores
Lucene, SOLR, ElasticSearch
Key/Value Stores
Hbase,Cassandra,
Riak, Dynamo
Document Stores
Marklogic, mongoDB, Cloudant, DocumentDB
©2015 Couchbase Inc. 7
Why migrate from RDBMS to NoSQL?
©2015 Couchbase Inc. 8
What Is Driving Use of NoSQL?
©2015 Couchbase Inc. 9
What’s the problem with relational?
Scalability and Performance
Single server Fixed capacity Under-provisioned Not enough resources Poor performance Over-provisioned Too many resources Unnecessary cost
©2015 Couchbase Inc. 10
How does NoSQL solve it?
Scalability and Performance
Cluster Commodity hardware Variable capacity Add resources On demand Without downtime High availability
©2015 Couchbase Inc. 11
Agility: Changing the data model
Relational database
• Modify the database schema• Modify the application code (e.g.
Java)• Modify the interface (e.g. HTML5/JS)
Document database
• Modify the interface (e.g. HTML5/JS)
©2015 Couchbase Inc. 12
Identifying the right application
Have one or more of the following characteristics or requirements:
Iterate faster Send and receive JSON Provide low latency at any throughput Support many concurrent users
Supports users anywhere and everywhere Be available 24x7 Store terabytes of data Read and write to multiple data centers
Service
RDMBS
Service Service
NoSQL
Application
©2015 Couchbase Inc. 13
Identifying the right application
Examples
• Product Catalog Service• Asset Tracking Service• Content Management Service
• Application Configuration Service • Customer Management Service• File or Streaming Metadata Service
Service
RDMBS
Service Service
NoSQL
Application
©2015 Couchbase Inc. 14
How can you introduce NoSQL with microservices?
Introduce NoSQL with a single service Could be a new service, could be an existing one
Update other services one at a time Where it makes sense…
Polyglot!
©2015 Couchbase Inc. 15
How can you introduce NoSQL with caching?
Deploy as mission-critical cache Primary source of engagement Handles all external reads & writes Results in less RDMBS costs (hardware) Results in less MIPS (mainframe usage) Leverage streaming replication to sync
©2015 Couchbase Inc. 16
Modeling you Data
©2015 Couchbase Inc. 17
Data Modeling
An ecommerce order in relational world
©2015 Couchbase Inc. 18
Data Modeling
Embed vs Refer
©2015 Couchbase Inc. 19
Data Modeling
Embed vs Refer
©2015 Couchbase Inc. 20
Modeling your data: Fixed vs. self-describing schema
©2015 Couchbase Inc. 21
Modeling your data: The flexibility of JSON
Same document type,Different fields
• Different types• Optional• On-demand
©2015 Couchbase Inc. 22
Modeling your data: Strategies and best practices
If… Then…
Relationship is one-to-one or one-to-many Store related data as nested objects
Relationship is many-to-one or many-to-many Store related data as separate documents
Data reads are mostly parent fields Store children as separate documents
Data reads are mostly parent + child fields Store children as nested objects
Data writes are mostly parent or child (not both) Store children as separate documents
Data writes are mostly parent and child (both) Store children as nested objects
©2015 Couchbase Inc. 23
What is Couchbase Server?
©2015 Couchbase Inc. 24
What’s Couchbase?
Couchbase is the company behind Couchbase Server & Couchbase Mobile
• Open source JSON database• Founded 2010• 500+ enterprise customers globally
Some of our customers:
Couchbase Server can be deployed as:
Document database Key-value store Distributed cache
©2015 Couchbase Inc. 25
Top Use Cases
Profile Management Personalization 360o Customer View Internet of Things
Content Management Catalog Real Time Big Data Digital Communication
Mobile Applications
High Availability Caching
©2015 Couchbase Inc. 26
What is Couchbase Server?
The Scalability of NoSQL + The Flexibility of JSON + The Power of SQL
Develop with Agility Operate at Any Scale
Elastic scalability Consistent high performance Always-on availability Multi-Data Center Deployment Simple & Powerful Administration Enterprise-grade Security
Easier, faster development Flexible data modeling Powerful querying & indexing SQL integration & migration Big data integration Mobile / IoT Extensions
[{
"brewery": "BrewDog Ltd","beers": 4,
},{
"brewery": "Founders Brewing""beers": 4,
},…
]
SELECT brewery.name AS brewery, count(*) AS beersFROM `beer-sample` beerINNER JOIN `beer-sample` breweryON KEYS beer.brewery_idWHERE beer.type = "beer" AND
beer.style = "American-Style Imperial Stout" ANDbrewery.type = "brewery"
GROUP BY brewery.nameHAVING count(*) > 2ORDER BY beers DESC, brewery ASC;
CREATE INDEX beer_styles ON `beer-sample`(style) WHERE type = "beer”;
©2015 Couchbase Inc. 28
Integrating RDBMS and NoSQL
©2015 Couchbase Inc. 29
The reality: The Data Layer Landscape
Challenges
Legacy Systems Different Technologies Data Ownership Bulk loading Data synchronization
RDBMSNoSQL
RDBMS
RDBMSNoSQL
NoSQL
NoSQL
HadoopMainframe
©2015 Couchbase Inc. 30
The reality: it is integration time!
Challenges
Legacy Systems Different Technologies Data Ownership Bulk loading Data synchronization
©2015 Couchbase Inc. 31
Integrating RDBMS and NoSQL
Tasks
Export/Import/Bulk loading Data synchronization Data Modeling: NoSQL- RDBMS gap
Tools
Export/Import/Bulk loading SDK Client API ETL Tools (Talend, Informatica, Pentaho,...)
RDBMS NoSQL
RDBMS
RDBMSNoSQL
NoSQL
NoSQL
HadoopMainframe
©2015 Couchbase Inc. 32
Integrating RDBMS and NoSQL: Demo 1
Demo: Bulk loading from Oracle to Couchbase
Custom tool based on SDK API Takes care of data types https://github.com/mahurtado/oracle2couchbase
©2015 Couchbase Inc. 33
Integrating RDBMS and NoSQL: Demo 1
Demo: Bulk loading from Oracle to Couchbase
©2015 Couchbase Inc. 34
Integrating RDBMS and NoSQL: Demo 2
Demo: Bulk loading from PostgreSQL to Couchbase
Use Case: Product Catalog Export to csv from PostgreSQL utility Import using Couchbase utility: cbbackupmgr
©2015 Couchbase Inc. 35
Integrating RDBMS and NoSQL: Demo 3
Demo: Synchronize data from Oracle to Couchbase
Using Oracle Golden Gate Golden Gate Java Adapter Using Couchbase SDK https://github.com/mahurtado/CouchbaseGoldenGateAdapter
©2015 Couchbase Inc. 36
Integrating RDBMS and NoSQL: Demo 4
Demo: Synchronize data from Couchbase to PostgreSQL
Using Couchbase Kafka connector Kafka consumer writes on PostgreSQL
Real Time changes
Couchbase Kafka
Connector
Kafka Consumer
Thank you!