no sql

RDBMS ?

Prateek Jain

12-Jul-2012

Solutions…

Web server Web server

App server App server

Web server

Cache server

RDBMS

CMS Data Feeds

Common Architecture

SQL - Story till now…

Stable environment. No more discussions on Data stores. Easy to train and employ people. SQL running effectively at core.

SQL - Story till now…

For dealing with lists (as tables) it’s a great language,dynamic and relatively fast

• Sure it has a few problems but give me a language that doesn’t

What Next…?

We need to fast, scale and be part of web

ORM - OMG!

The effort of trying to convert something inherently hierarchical into something relational

Probably the biggest waste of programming time, lines of code and source of bugs and latency is ORM

Challenges

Data grows exponentially. Data is unstructured. Data is huge and spread across 100’s/1000’s

of nodes. SQL is useful - when things are flat

Lots of data

In the banking world we have a lot of data Today 50-100,000 quotes a second isn’t

unusual It gets more complex...

• 10,000 portfolios, each with 1,000 buy/sell orders at specific prices

• We now have 100,000 prices coming in every second and 10 million orders to watch

Time is critical

In the world of trading only the first one gets the deal, there is no second place.

While being first to have the order is what makes the money banks now have a “new” problem

“RISK”

Lots of data, lots of calculations

There are two main flavors of distributed computing• Data• Computation

Often they are closely related but not always. To achieve either we usually need lots of memory and CPUs We don’t stack them or put them in clusters these days, we

distribute them

Why not RDBMS?

Not designed to scale out. Strongly ACID complaint. Slower running queries (specially in joins). Schema based. Not suited for changing data structure.

CAP Theorem

C – consistency A – availability P – partition tolerance

** You must make trade-offs and sacrifice at least one in favor of the other two.

NoSql

Not Just Sql

Categories

Document BasedDocument Based

Column BasedColumn Based

Key/Value BasedKey/Value Based

Graph BasedGraph Based

Data Structure BasedData Structure Based

Example ProductsExample Products

Eventual Consistency

Given a sufficiently long period of time, over which no updates are sent, one can expect that all updates will, eventually, propagate through the system and all the replicas will be consistent.

In the presence of continuing updates, an accepted update eventually either reaches a replica or the replica retires from service.

Eventual Consistency

Scalability

Scalability

Scalability is the ability of a system to increase throughput with addition of resources to address load increases.

Scalability can be achieved by:– Provisioning a large and powerful resource to meet the additional

demands. – It can be achieved by relying on a cluster of ordinary machines to

work as a unit.

How to choose ?

Scalability Transactional integrity and consistency Data modeling Query support Access and interface availability

Scalability

column-family-centric NoSQL databases are a good choice if extreme scalability is a requirement.

Not well suited for real-time transaction processing. (RDBMS is best)

Eventually consistent NoSQL options, like Cassandra or Riak, may be workable.

Transactional Integrity and Consistency

Batch-centric analytics on warehoused data is also not subject to transactional requirements.

Data sets that are written once for e.g., web traffic log files, social networking status updates, advt. click-through imprints, road-traffic data, stock market tick data, game scores etc.

Transactional Integrity and Consistency

If range operations are common and integrity of updates is required, an RDBMS is the best choice.

If atomicity at an individual item level is sufficient, then column-family databases, document databases.

Data Modeling

RDBMS offers a consistent way of modeling data. Relational algebra underlies the data model.

In the NoSQL world there is no such standardized and well-defined data model.

Data Modeling

If relaxed schema is your primary reason for using NoSQL, then MongoDB is a great option for getting started with NoSQL.

MongoDB is used by many web-centric businesses.

Querying Support

An RDBMS thrives on SQL support, which makes accessing and querying data easy.

Among document databases, MongoDB provides the best querying capabilities.

For key/value pairs and in-memory stores, nothing is more feature-rich than Redis as far as querying capabilities go.

Querying Support

Column-family stores like HBase have little to offer as far as rich querying capabilities go.

Project called Hive makes it possible to query HBase using SQL-like syntax and semantics.

Access and Interface Availability

MongoDB has the notion of drivers. CouchDB always has the RESTful HTTP

interface available. Redis, Membase, Riak, HBase, Hypertable,

Cassandra, and Voldemort have support for language bindings to connect from most mainstream languages.

Performance

50/50 Read and Update

Results showthat under this test case Apache Cassandra outperforms the competition on both read and update latencies.

HBase comes close but stays behind Cassandra.

95/5 Read and Update

The sorted ordered column-family stores perform best for contiguous range reads.

HBase seems to deliver consistent performance for reads, irrespective of the number of operations per second.

MySQL delivers the best performance for read-only cases.

Future?

Coexistence

Future

Getting ready for polyglot persistence. Understanding the database technologies

suitable for immutable data sets. Choosing the right database to facilitate ease

of application development.

Examples

Linked In uses Hadoop for many large-scale analytics jobs like probabilistically predicting people you may know.

Facebook (mysql + HBase, cassandra, ZooKeeper) Twitter (mysql + Cassandra + FlockDB)

Questions?

Feedback

trainer.prateek@gmail.com