migration and coexistence between relational and nosql databases by manuel hurtado

Migration and Coexistence between RDBMS and NoSQL databases

Manuel HurtadoSolutions Engineer


What is NoSQL?


DynamoOctober 2007

CassandraAugust 2008

BigtableNovember 2006

VoldemortFebruary 2009

Very few organizations want to (fewer can) build and maintain database software technology.But every organization participating in the digital economy needs this technology.

Why NoSQL

New data solutions to address these needs: No schema required before inserting data No schema change required to change data format Auto-Sharding without application participation Distributed queries Managed Cache Data Synchronization ( multi-datacenter)


What is NoSQL?

No SQL? No. Not only SQL? Not really. Non relational? Yes.

Distributed (most)– Scaled out, not up

• Elasticity and commodity hardware

– Partitioned and replicated• Scalability, performance, availability

Schema-less (most)– Flexible model– JSON (some)


World of Operational Databases

6

Graph Stores

Neo4j

RDBMS/Transactional Stores

Oracle, DB2, SQLServer, MySql, Postgres

Caching Stores

Coherence, Redis, Gemfire

Full-text Stores

Lucene, SOLR, ElasticSearch

Key/Value Stores

Hbase,Cassandra,

Riak, Dynamo

Document Stores

Marklogic, mongoDB, Cloudant, DocumentDB


Why migrate from RDBMS to NoSQL?


What Is Driving Use of NoSQL?


What’s the problem with relational?

Scalability and Performance

Single server Fixed capacity Under-provisioned Not enough resources Poor performance Over-provisioned Too many resources Unnecessary cost


How does NoSQL solve it?

Scalability and Performance

Cluster Commodity hardware Variable capacity Add resources On demand Without downtime High availability


Agility: Changing the data model

Relational database

• Modify the database schema• Modify the application code (e.g.

Java)• Modify the interface (e.g. HTML5/JS)

Document database

• Modify the interface (e.g. HTML5/JS)


Identifying the right application

Have one or more of the following characteristics or requirements:

Iterate faster Send and receive JSON Provide low latency at any throughput Support many concurrent users

Supports users anywhere and everywhere Be available 24x7 Store terabytes of data Read and write to multiple data centers

Service

RDMBS

Service Service

NoSQL

Application


Identifying the right application

Examples

• Product Catalog Service• Asset Tracking Service• Content Management Service

• Application Configuration Service • Customer Management Service• File or Streaming Metadata Service

Service

RDMBS

Service Service

NoSQL

Application


How can you introduce NoSQL with microservices?

Introduce NoSQL with a single service Could be a new service, could be an existing one

Update other services one at a time Where it makes sense…

Polyglot!


How can you introduce NoSQL with caching?

Deploy as mission-critical cache Primary source of engagement Handles all external reads & writes Results in less RDMBS costs (hardware) Results in less MIPS (mainframe usage) Leverage streaming replication to sync


Modeling you Data


Data Modeling

An ecommerce order in relational world


Data Modeling

Embed vs Refer


Modeling your data: Fixed vs. self-describing schema


Modeling your data: The flexibility of JSON

Same document type,Different fields

• Different types• Optional• On-demand

Moderador

Notas de la presentación

KEY POINT: YOU HAVE THE OPTION TO REPRESENT DATA QUITE DIFFERENTLY USING JSON AS OPPOSED TO A RELATIONAL DATABASE. - Where in relational databases you might have to have multiple tables to best represent your data, in JSON you can model your data like an object might already be in your programming language of choice. No ORM (Object Relational Model) needed. You can do relationship in Couchbase, but they are different than in a relational database and outside of the scope of an intro call normally. Make sure to stress that normalization is still something that can be done in Couchbase where it makes sense for the application, but this diagram is something that helps people coming from relational understand what is possible for JSON.


Modeling your data: Strategies and best practices

If… Then…

Relationship is one-to-one or one-to-many Store related data as nested objects

Relationship is many-to-one or many-to-many Store related data as separate documents

Data reads are mostly parent fields Store children as separate documents

Data reads are mostly parent + child fields Store children as nested objects

Data writes are mostly parent or child (not both) Store children as separate documents

Data writes are mostly parent and child (both) Store children as nested objects

Moderador


What types of relationships are being modeled? How are the relationships accessed?


What is Couchbase Server?


What’s Couchbase?

Couchbase is the company behind Couchbase Server & Couchbase Mobile

• Open source JSON database• Founded 2010• 500+ enterprise customers globally

Some of our customers:

Couchbase Server can be deployed as:

Document database Key-value store Distributed cache


Top Use Cases

Profile Management Personalization 360o Customer View Internet of Things

Content Management Catalog Real Time Big Data Digital Communication

Mobile Applications

High Availability Caching

Moderador


Key Point: Enterprises are using Couchbase across a range of mission critical use cases. As the slide shows, Couchbase supports a wide range of use cases, from Profile Management to Fraud Detection. Each use case has its own set of requirements – some need very high performance, some need very high availability, some need flexibility of the data model. The ability to meet all of these requirements is what has driven adoption of Couchbase. Profile Management - Maintain user profiles to support authentication, preferences, and more over multiple channels. Personalization - Ingest clickstream data at high throughput to create and refine millions of visitor profiles. Customer 360° View - Aggregate customer information from multiple sources with different data models. Internet of Everything - Collect high volume, high velocity sensor and telemetry data from industrial and consumer devices. Mobile Applications - Build mobile apps with offline support via an embedded database and automatic synchronization. Content Management - Manage a variety of static and dynamic content without the limitations of a fixed data model. Catalogs - Engage customers by providing low latency access to product or service catalogs, online or in-store. Real-Time Big Data - Leverage streaming integration with Hadoop and Storm to support and enable real-time analytics. Digital Communication - Support real-time interaction and communication with low latency read/write access to messages. High Availability Caching - Deploy Couchbase Server as standardized distributed caching layer Compatible with memcached, highly optimized for latency and throughput Shared nothing, replicated and persistent for reliability Support for JSON as well as any binary data type Shared-nothing, replicated and persistent architecture


What is Couchbase Server?

The Scalability of NoSQL + The Flexibility of JSON + The Power of SQL

Develop with Agility Operate at Any Scale

Elastic scalability Consistent high performance Always-on availability Multi-Data Center Deployment Simple & Powerful Administration Enterprise-grade Security

Easier, faster development Flexible data modeling Powerful querying & indexing SQL integration & migration Big data integration Mobile / IoT Extensions

[{

"brewery": "BrewDog Ltd","beers": 4,

},{

"brewery": "Founders Brewing""beers": 4,

},…

]

SELECT brewery.name AS brewery, count(*) AS beersFROM `beer-sample` beerINNER JOIN `beer-sample` breweryON KEYS beer.brewery_idWHERE beer.type = "beer" AND

beer.style = "American-Style Imperial Stout" ANDbrewery.type = "brewery"

GROUP BY brewery.nameHAVING count(*) > 2ORDER BY beers DESC, brewery ASC;

CREATE INDEX beer_styles ON `beer-sample`(style) WHERE type = "beer”;


Integrating RDBMS and NoSQL


The reality: The Data Layer Landscape

Challenges

Legacy Systems Different Technologies Data Ownership Bulk loading Data synchronization

RDBMSNoSQL

RDBMS

RDBMSNoSQL

NoSQL

NoSQL

HadoopMainframe


The reality: it is integration time!

Challenges

Legacy Systems Different Technologies Data Ownership Bulk loading Data synchronization


Integrating RDBMS and NoSQL

Tasks

Export/Import/Bulk loading Data synchronization Data Modeling: NoSQL- RDBMS gap

Tools

Export/Import/Bulk loading SDK Client API ETL Tools (Talend, Informatica, Pentaho,...)

RDBMS NoSQL

RDBMS

RDBMSNoSQL

NoSQL

NoSQL

HadoopMainframe


Integrating RDBMS and NoSQL: Demo 1

Demo: Bulk loading from Oracle to Couchbase

Custom tool based on SDK API Takes care of data types https://github.com/mahurtado/oracle2couchbase

https://github.com/mahurtado/oracle2couchbase



Demo: Bulk loading from Oracle to Couchbase



Demo: Bulk loading from PostgreSQL to Couchbase

Use Case: Product Catalog Export to csv from PostgreSQL utility Import using Couchbase utility: cbbackupmgr



Demo: Synchronize data from Oracle to Couchbase

Using Oracle Golden Gate Golden Gate Java Adapter Using Couchbase SDK https://github.com/mahurtado/CouchbaseGoldenGateAdapter

https://github.com/mahurtado/CouchbaseGoldenGateAdapter



Demo: Synchronize data from Couchbase to PostgreSQL

Using Couchbase Kafka connector Kafka consumer writes on PostgreSQL

Real Time changes

Couchbase Kafka

Connector

Kafka Consumer

Thank you!

migration and coexistence between relational and nosql databases by manuel hurtado

Technology