big data: from mammoth to elephant

Roman Nikitchenko, 10.05.2015

BIG DATA: FROM MAMMOTH TO ELEPHANT

MAMMOTHThe only real truth we know about them is their rests. Do you feel your enterprise data infrastructure goes this way?

Come and see in the nearest data center...

2

TWO YEARS AGO● Our exciting high scalability realtime

BIG DATA solution with broad technologies stack in production.

3

This is our PRESENT DAY

.. yet is powered by

4

storage storage

SQL DB Processed inbound data

Inbound Outbound


Healthcare providers data: labs, cares ...

Mostly insurance companies

SQL DB Application data

SQL DB Outbound information

OUR INITIAL STATE: TOP VIEW

CLIENT APPLICATIONS

CLIENT APPLICATIONS

CLIENT APPLICATIONS

CLIENT APPLICATIONS

5

storage storage


Inbound Outbound






CLIENT APPLICATIONS

CLIENT APPLICATIONS

CLIENT APPLICATIONS

CLIENT APPLICATIONS

Inbound data archives(pretty short cycle)

One SQL DB per application

Huge amount of data. Serious amount of duplicates

How about retention and data issues investigation?


6

Outbound flow is slow because of RDBMS processing

storage storage


Inbound Outbound






CLIENT APPLICATIONS

CLIENT APPLICATIONS

CLIENT APPLICATIONS

CLIENT APPLICATIONS

Inbound data retention cycle is short, so prolonged period data investigation is hard

Overall huge amount of SQL databases, high operational complexity

One application DB per service client makes inter-application analytics and monitoring extremely hard

YELLOW ALARMS


7

BIG DATA

Better ways to store huge data volumes: cheaper, safer and easier.

WHAT TO RUN FOR?

MORE STORAGE

9

BIG DATAWHAT TO RUN FOR?

Scalable effective distributed processing models to open new opportunities like machine learning.

MORE POWER

10

BIG DATAWHAT TO RUN FOR?

More flexible data structures closer to subject area and real world.

11

RDBMS LIMITS● Good for anything

● Not so good for anything in particular

OUR MAIN ENEMY WAS ...

12

MASSIVE ANALYSISIs about massive access to your data objects

Yourdatabase

Subject area objects data




Processing

Processing

Processing

Processing

Transformation from database structure into object structure

Distributed parallel

processing

Effective results collection

Distributed processing

results to be joined

WHY SQL IS EVIL

13

RDBMS LIMITS

When you go massive processing, objects collection is getting too complex. Think about 100.000.000 people data scan.

Address ID City Street

1 New York 1020, Blue lake

2 Atlanta 203, Bricks av.

3 Seattle 120, Green drv.

FirstName LastName Address Payer

John Smith 1 2

Kate Davis 2 1

Samuel Brown 3 2

Payer ID Name State

1 SaferLife GA

2 YourGuard CA

Kate Davis,Atlanta 203, Bricks av.SafeLife, GA

SUBJECT AREA OBJECT COLLECTION

14

FirstName

LastName

Address

Payer

Birthday

RDBMS LIMITSFirstName LastName Address Payer

John Smith 1 2

Kate Davis 2 1

Samuel Brown 3 2

And now let us add new «Birthday» column.Easy as pie!

Let it be Patients table ...

ALTER TABLE Patient ADD Birthday ...

TABLE STRUCTURE MODIFICATION

Let's do this with 2.000.000.000 rows MySQL table in production. What to do if your table grows further?

15

ANY RELATIONAL DATA MODELSOONER OR LATER

16

Your SQLdatabase

Shard

Shard

Shard

Shard

Processing

Processing

Processing

Processing

How to partition data? What to do

when new shard is added?

Need another cluster for

processing?


results to be joined

HOW TO SCALE?

RDBMS LIMITS

17

If you need to store plain text log, collection of objects for a long time or current user session attributes do you really need SQL?

18

Cross-application data storage




Small realtime requests Batch analytic and reporting

load

ETL

ETL

ETL

● One-time ETL as initial step and backup strategy.● Full migration to Apache Hbase.● As a transition period solution — realtime synchronization.

OUR INITIALBIG PLAN WAS

19

OPEN SOURCE framework for big data. Both distributed storage and processing

Provides RELIABILITY and fault tolerance by SOFTWARE design (for example file system with replication factor 3 as default one.Horizontal scalability from

single computer up to thousands of nodes

Why Hadoop (initially 1.x)?

20

First ever worldDATA OS

10.000 nodes computer... Can start in production from just 4 servers, 1 of them is for management and coordination. Single server is enough for development environment.

21

HBase motivationWHY

LATENCY, SPEED AND ALL HADOOP PROPERTIES

22

Database

Region server


WHY YET ?

DataNode Node

File system Hardware

TaskTracker

Region server DataNode NodeTaskTracker



● Good both for OLTP and batch load.● Natural scaling and reliability with Hadoop.● Data processing locality, natural sharding with regions.● Coordination with ZooKeeper.

23

ZooKeeperBecause coordinating distributed systems is a Zoo.

● Quorum based service for fast distributed system coordination.

● Came in our stack with Apache Hbase where it was needed for coordination. Now is part of core Hadoop infrastructure.

● Yet we use it for our own applications,

24

Finally we went initial production with HADOOP 2.0

RESOURCE MANAGEMENT

DISTRIBUTED PROCESSING

FILE SYSTEM

COORDINATION

HADOOP 2.x CORE

25

Database

Region server

Distributed processing & coordination

Real initial approach

DataNode Node

File system Hardware

Region server DataNode Node



● ZooKeeper Instances are distributed among cluster.● MapReduce is not service in Hadoop 2.x, just YARN application.

Resource management

NodeManager

NodeManager

NodeManager

NodeManager

26

FIRST REAL RESULT

Cross-application data storage




Small realtime requests Batch analytic and reporting

load

ETL

ETL

ETL

CLOSE BUT NOT EXACT PLAN

Daily ETL. Satisfied our daily reporting needs with major SQL infrastructure offload. Direct profit — massive processing is much faster, can handle inter-application data.

DO NOT WEAR PINK GLASSES

27

APPROACH WE HAVE FIXED MUCH LATER

SQLserver

JOIN

Table1

Table2

Table3

Table4

ETL stream

SQLserver

JOIN

Table1

Table2

Table3

Table4

ETL stream

ETL stream

ETL stream

ETL stream

BIG DATA shard

BIG DATA shard

BIG DATA shard

BIG DATA shard

BIG DATA shard

BIG DATA shard

Bulkload

Bulkload

28

Hadoop: don't do it yourself

DON'T DO IT YOURSELF

Because of number of factors starting from our distributed team support needs we have selected

29

x MAX+

=

BIG DATA

BIG DATA

BIG DATA

HADOOP as INFRASTRUCTURE

30

WHERE TO GO FROM HERE?31

The admission of temporary residents into Canada is a privilege, not a right.

http://www.cic.gc.ca/

SEARCH / SECONDARY INDICES

32

NO SEARCH OUT OF THE BOX OTHER THAN LINEAR SCAN OVER THE TABLE AND FILTERS.


The same happened to be applicable to secondary indices in Hbase.

33


HOW WE MADE IT

HBase handles user data changes

Indexes are built on SOLR

NGData Lily indexer transforms data

changes into SOLR index updates

34

HBase: Data and search integration

Data update

Client

User just puts (or deletes) data.

Search responses

Lily HBase NRT indexerREPLICATION

Translates data changes into SOLR

index updates.

SOLR cloud

Search requests (HTTP)

Apache Zookeeper does all coordination Provides real

indexing

Search and indexing together

35

● Kafka is a high throughput distributed messaging system.

● Allows true realtime system reaction through publish-subscribe approach.

● New services can subscribe to data events stream.

GOING REALTIMEBatch load

Realtime load

New data

36

● Kafka can be separated from Hadoop infrastructure or have backup cluster.

● Data publishers can switch to another cluster.

● Subscribers (including Spark on Hadoop) keep 2 places of subscription.

● So now you are free to put Kafka cluster in maintenance or backup subscribers.

GOING REALTIME

GENTLY

MAINTENANCE

37

This is our PRESENT DAY

.. yet is powered by

38

SO WHERE ARE WE GOING?

39

OVER BIG DATAREACTIVE MANIFESTO

MOTIVATION

… users expect millisecond response times and 100% uptime. Data is measured in Petabytes. Today's demands are simply not met by yesterday’s software architectures.

40


… we want systems that are Responsive, Resilient, Elastic and Message Driven. We call these Reactive Systems. http://www.reactivemanifesto.org/

41

http://www.reactivemanifesto.org/


Responsiveness is the cornerstone of usability and utility, but more than that, responsiveness means that problems may be detected quickly and dealt with effectively.

RESPONSIVE

42

OVER BIG DATAREACTIVE MANIFESTOThe system stays responsive in the face of failure.

… The client of a component is not burdened with handling its failures.

RESILIENT All services here are located through ZooKeeper which is quorum based so resilience is achieved

43


Reactive Systems can react to changes in the input rate by increasing or decreasing the resources allocated to service these inputs.

ELASTICBoth HDFS and Hbase

allow dynamic node addition / removal

YARN already handles most resource allocation

work and makes progress

44


Reactive Systems rely on asynchronous message-passing to establish a boundary between components that ensures loose coupling.

MESSAGE DRIVEN

Asynchronous messages from

applications

Any application can subscribe, not only Hadoop services

45

LESSONS LEARNED

● No transition in one step. You enter Big Data world step by step.

● Change you mind first. You should stop thinking in old style. Do not try simply to map your existing approaches.

● No silver bullet. Don't ruin your existing infrastructure. Extend it. NoSQL is not always good and some cases are really to be kept on SQL. Use the right tool.

● As you progress you pay more attention to operations and reactive system properties.

46

QUESTION?

47

big data: from mammoth to elephant

Technology