where is data going? - rmdc keynote

© 2016 MapR Technologies 1© 2014 MapR Technologies

Where is Data Going?

Ted Dunning Chief Application Architect

© 2016 MapR Technologies 2

My Background• University, Startups

– Aptex, MusicMatch, ID Analytics, Veoh– big data since before it was big

• Open source– even before the internet– Calcite, Datafu, Drill, Kylin, Mahout, Myriad, Samoa, Singa, Storm,

Zookeeper– bought the beer at first HUG

• VP Incubator at Apache Software Foundation• MapR Chief Application Architect


When you see this

Real tech is coming shortly


IT Spending at an Inflection Point

Source: IDC, Gartner; Analysis & Estimates: MapRNext-gen consists of cloud, big data, software and hardware related expenses

2013 2014 2015 2016 2017 2018 2019 2020

(100,000)

(50,000)

-

50,000

100,000

150,000 Next Gen vs. Legacy Shrinkage ($M)

$120

100

80

60

40

20

(20)

(40)

(60)

(80)

(100)

In Billions

Total $ Growth of IT Market Next-Gen Growth Legacy Market Growth/Shrink in $


You Need to Make a Decision Now

In 4 years 90% of Data will be on next-gen technology


MapR Converged Platform

Enterprise Hadoop + Storage2011-2012

2013

2014

2015

2016

+ NoSQL

+ Interactive SQL

+ Global Event Streaming

+ …

Top Ranked Hadoop

Top Ranked NoSQL

Top Ranked SQL


Innovatio

n

$$

$

Reduce

Costs


Wait a minute!

What’s really going on?


Evolution of Data Storage

FunctionalityCompatibility

Scalability

LinuxPOSIX

Over decades of progress,Unix-based systems have set the standard for compatibility and functionality



Scalability

LinuxPOSIX

HadoopHadoop achieves much higher scalability by trading away essentially all of this compatibility





Scalability

LinuxPOSIX

Hadoop

MapR enhanced Apache Hadoop by restoring the compatibility while increasing scalability and performance


Scalability

POSIX



Scalability

LinuxPOSIX

Hadoop


Adding tables and streams enhances the functionality of the base file system


What is Convergence?

Files StreamsTables



Files StreamsTables


Basic System Design


Consumer Behavior


Reduce

Costs

$$$

Innovatio

n

© 2016 MapR Technologies 24PEOPLE

1.2BPEOPLE

Largest Biometric Database in the World


4 data centers

Synchronous replication EW

Asynchronous replication NS


EW distance is short


NS separation is large


Summary• Updates can happen anywhere, anytime

• Consistency maintained by insertion timestamp

• System is robust to link, power, data-center failure


Tenants Share Access to Data


Streaming and Micro-services


And just a slight bit of future tense here


Financial Services Use Case• For reference assume:

– 1000 - 10,000 unique senders and receivers– each bid or offer includes 10 recipients on average – bids and offers arrive at a rate of 300k messages / second

• What would you do?


Financial Services: Our Solution


Massive IoT• 100 million cars• 2kB / second

• Roaming across data centers


Streaming First for Micro-Services• How do applications need to change?

• How can we get micro-service benfits (this time around)?

• How does our development need to change?


Traditional Solution


What Happens Next?


How to Get Service Isolation


New Uses of Data


Data Platform

Integrated Analytics

Enterprise Grade

Scale

Legacy Support

Transformational Applications

Real Time


Let’s Review the Options

S T O R A G E

StorageIcon D A T A 0 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 1

0 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 1


H A D O O P


D T A

C O M P U T E

D A T A 0 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 10 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 1



C O M P U T E

S P A R K

D A T A



C A S S A N D R A

D T A

C O M P U T E

D A T A 0 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 10 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 1



C O M P U T E

D T AD A T A 0 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 10 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 1

Write In:–– –– –– –

O T H E R


C O N V E R G E D D A T A P L A T F O R M


Data in MotionData at Rest

C O M P U T E

D T AD A T A 0 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 10 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 1




Data in MotionData at Rest




CloudOn Premise




Legacy Next Generation




Real TimeBatch


Platform Appeal

Reduce CostsDrive Innovation

Graphic showing business users,


Platform Appeal for Architects and Administrators

1. Simplifies administration

2. Avoids cluster sprawl

3. Unifies security, data protection, disaster recovery


Platform Appeal for Developers

Flexibility Agility Simplicity Real Time


Integration of data-in-motion and data-at-rest supports continuous, low latency processing

Converged Data Platform

mapr.com/appblueprint


Q & A@mapr maprtech

[email protected]

Engage with us!

MapR

maprtech

mapr-technologies

where is data going? - rmdc keynote

Technology