Upload File

more than data

88
MORE THAN DATA Using Riak Core to Manage Distributed Services

Upload: milesoc

Post on 08-Jul-2015

286 views

Category:

Technology


0 download

Report

Tags:

DESCRIPTION

Slides for a talk given at RICON West 2013. Video is available at the very beginning of Day 1, Track 2 here: http://ricon.io/west2013.html Full edited videos will be available at a later date. This talk is about how StackMob uses Riak Core for distributing services.

TRANSCRIPT

Page 1: More than Data

MORE THAN DATAUsing Riak Core to Manage Distributed

Services

Page 2: More than Data

Who Am I?

Miles O’Connell

@milesoc (twitter/github)

Software Engineer at StackMob

Page 3: More than Data

Backend-As-A-Service

• Data API

• Push

• Third Party Marketplace

• etc.

Page 4: More than Data

Backend-As-A-Service

• Data API

• Push

• Third Party Marketplace

• etc.

Service Oriented Architecture (SOA)

Page 5: More than Data

Service Oriented Architecture

Creative Commons License – Flickr/Jared Tarbell

Page 6: More than Data

CAP Theorem

• Consistency

• Availability

• Partition Tolerance

Page 7: More than Data

Service Oriented Architecture

• Versioning (“Consistency”)

• Availability

• Partition Tolerance

Page 8: More than Data

Service Oriented Architecture

Service A Service B

Service C Service D

Page 9: More than Data

Service Oriented Architecture

Service A Service B

Service C Service D

Service B

(v2)

Service D

(v2)

Page 10: More than Data

Service Oriented Architecture

Service A Service B

Service C Service D

Service B

(v2)

Service D

(v2)

Page 11: More than Data

Service Oriented Architecture

Service A

Service C

Service B

(v2)

Service D

(v2)

Page 12: More than Data

Service Oriented Architecture

• Versioning (“Consistency”)

• Availability

• Partition Tolerance

Page 13: More than Data

Custom Code

Data API Push API Versioning

3rd Partyetc.

etc.etc.

Page 14: More than Data

Custom Code

Data API Push API Versioning

3rd Partyetc.

etc.etc.

Custom Code

Page 15: More than Data

Custom Code

Data API Push API Versioning

3rd Partyetc.

etc.etc.

Page 16: More than Data

Service Oriented Architecture

• Versioning (“Consistency”)

• Availability

• Partition Tolerance

Page 17: More than Data

Service Oriented Architecture

• Versioning (“Consistency”)

• Availability

• Partition Tolerance

• Automated

Page 18: More than Data

Riak Core

• https://github.com/basho/riak_core

• Erlang library for building distributed systems

https://github.com/basho/riak_core
https://github.com/basho/riak_core
Page 19: More than Data

Riak Core2160 0

Page 20: More than Data

Riak Core2160 0

Key

Page 21: More than Data

Riak Core

vnode

Page 22: More than Data

Riak Core

Physical Server 1

Physical Server 2

Physical Server 3

Page 23: More than Data

Riak Core

Physical Server 1

Physical Server 2

Physical Server 3

Page 24: More than Data

Riak Core

K, V

Physical Server 1

Physical Server 2

Physical Server 3

Page 25: More than Data

Riak Core

K, V

Physical Server 1

Physical Server 2

Physical Server 3

Page 26: More than Data

Riak Core

K, V

Physical Server 1

Physical Server 2

Physical Server 3

Page 27: More than Data

Riak Core

K, V

Physical Server 1

Physical Server 2

Physical Server 3

Page 28: More than Data

Riak Core

K, V

Physical Server 1

Physical Server 2Physical Server 3

Page 29: More than Data

Riak Core

K, V

Physical Server 1

Physical Server 2

Physical Server 3

Page 30: More than Data

Riak Core

K, V

Physical Server 1

Physical Server 2

Physical Server 3

Page 31: More than Data

SOA on Riak Core

RequestService

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 32: More than Data

SOA on Riak Core

RequestCC

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 33: More than Data

SOA on Riak Core

RequestCC

Instance

Physical Server 1

Physical Server 2

Page 34: More than Data

SOA on Riak Core

Request

CC

Instance

Physical Server 1

Physical Server 2

Page 35: More than Data

SOA on Riak Core

Request

Webmachine

Physical Server 1

Physical Server 2

CC

Instance

Page 36: More than Data

SOA on Riak Core

Request

Webmachine

proxy_coord

Physical Server 1

Physical Server 2

CC

Instance

Page 37: More than Data

SOA on Riak Core

Request

Webmachine

proxy_coord

vnode

Physical Server 1

Physical Server 2

CC

Instance

Page 38: More than Data

SOA on Riak Core

Request

Webmachine

proxy_coord

vnode

instance_mgr

Physical Server 1

Physical Server 2

CC

Instance

Page 39: More than Data

SOA on Riak Core

Request

Webmachine

proxy_coord

vnode

Physical Server 1

Physical Server 2

proxy_op

instance_mgr

CC

Instance

Page 40: More than Data

SOA on Riak Core

Request

Webmachine

proxy_coord

vnode

Physical Server 1

Physical Server 2

proxy_op

instance_mgr

CC

Instance

Page 41: More than Data

SOA on Riak Core

Request

Webmachine

proxy_coord

vnode

Physical Server 1

Physical Server 2

proxy_op

instance_mgr

Deploy

deploy_coord

deploy_op

instance_mgr

CC

Instance

CC

Instance

Page 42: More than Data

SOA on Riak Core

RequestCC

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 43: More than Data

Service Oriented Architecture

• Versioning (“Consistency”)

• Availability

• Partition Tolerance

• Automated

Page 44: More than Data

SOA on Riak Core - Replication

RequestService

Instance

Service

Instance

Service

Instance

Page 45: More than Data

SOA on Riak Core - Replication

• R

• How many replicas must we read

• W

• How many replicas must we write

• N

• How many replicas should exist

Page 46: More than Data

Riak Core - Replication

K, V

Page 47: More than Data

Riak Core - Replication

K, V

Physical Server 1

Physical Server 2

Physical Server 3

Page 48: More than Data

SOA on Riak Core - Replication

Request

Physical Server 1

Physical Server 2

Physical Server 3

Service

Instance

Service

Instance

Service

Instance

Page 49: More than Data

SOA on Riak Core - Replication

RequestService

Instance

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Service

Instance

Page 50: More than Data

SOA on Riak Core - Replication

RequestService

Instance

Service

Instance

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 51: More than Data

SOA on Riak Core - Replication

Request

Service

Instance

Service

Instance

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 52: More than Data

SOA on Riak Core - Replication

Request

Service

Instance

Service

Instance

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 53: More than Data

SOA on Riak Core - Replication

Request

Service

Instance

Service

Instance

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 54: More than Data

SOA on Riak Core - Replication

Request

Service

Instance

Service

Instance

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 55: More than Data

SOA on Riak Core - Replication

Deploy

Physical Server 1

Physical Server 2

Physical Server 3

Page 56: More than Data

SOA on Riak Core - Replication

Deploy

Service

Instance

Service

Instance

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 57: More than Data

SOA on Riak Core

• Versioning (“Consistency”)

• Availability✔

• Partition Tolerance

Page 58: More than Data

SOA on Riak Core

• Versioning (“Consistency”)

• Availability✔

• Partition Tolerance

Page 59: More than Data

SOA on Riak Core - Handoff

Service

Instance

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 60: More than Data

SOA on Riak Core - Handoff

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 61: More than Data

SOA on Riak Core - Handoff

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 62: More than Data

SOA on Riak Core - Handoff

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 63: More than Data

SOA on Riak Core - Handoff

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 64: More than Data

SOA on Riak Core - Handoff

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 65: More than Data

SOA on Riak Core - Handoff

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 66: More than Data

SOA on Riak Core - Handoff

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Service

Instance

Page 67: More than Data

SOA on Riak Core - Handoff

Service

Instance

Service

Instance

Physical Server 1

Physical Server 2

Physical Server 3

Page 68: More than Data

SOA on Riak Core

• Versioning (“Consistency”)

• Availability✔

• Partition Tolerance✔

Page 69: More than Data

SOA on Riak Core

• Versioning (“Consistency”)

• Availability✔

• Partition Tolerance✔

Page 70: More than Data

SOA on Riak Core - Versioning

Deploy

(v2)

Service

(v1)

Service

(v1)

Service

(v1)

Page 71: More than Data

SOA on Riak Core - Versioning

Deploy

(v2)

Service

(v1)

Service

(v1)

Service

(v1)

Page 72: More than Data

SOA on Riak Core - Versioning

/{service}/{version}/api/{endpoint}

Page 73: More than Data

SOA on Riak Core - Versioning

Request

Service

(v1)

Service

(v1)

Service

(v1)

Bastion

Page 74: More than Data

SOA on Riak Core - Versioning

Service B

(v1)

Service B

(v1)

Service A

(v1)

Service A

(v1)

Page 75: More than Data

SOA on Riak Core - Versioning

Service B

(v1)

Service B

(v1)

Service A

(v1)

Service A

(v1)

Page 76: More than Data

SOA on Riak Core - Versioning

Service B

(v1)

Service B

(v1)

Service A

(v1)

Service A

(v1)

Service A

(V2)

Service A

(V2)

Service B

(V2)

Service B

(V2)

Page 77: More than Data

SOA on Riak Core - Versioning

Service B

(v1)

Service B

(v1)

Service A

(v1)

Service A

(v1)

Service A

(V2)

Service A

(V2)

Service B

(V2)

Service B

(V2)

Page 78: More than Data

SOA on Riak Core - Versioning

Service B

(v1)

Service B

(v1)

Service A

(v1)

Service A

(v1)

Service A

(V2)

Service A

(V2)

Service B

(V2)

Service B

(V2)

Page 79: More than Data

SOA on Riak Core

• Versioning (“Consistency”) ✔?

• Availability✔

• Partition Tolerance✔

Page 80: More than Data

SOA on Riak Core - Integration

Three key features

Page 81: More than Data

SOA on Riak Core - Integration

Three key features

• Management API (Startup/Shutdown)

Page 82: More than Data

SOA on Riak Core - Integration

Three key features

• Management API (Startup/Shutdown)

• Consistent Service Responses

Page 83: More than Data

SOA on Riak Core - Integration

Three key features

• Management API (Startup/Shutdown)

• Consistent Service Responses

• Status Check Endpoint

Page 84: More than Data

SOA on Riak Core

• Riak Core can work for any distributed system

Page 85: More than Data

SOA on Riak Core

• Riak Core can work for any distributed system

• Riak Core provides:

• Availability

• Partition Tolerance

Page 86: More than Data

SOA on Riak Core

• Riak Core can work for any distributed system

• Riak Core provides:

• Availability

• Partition Tolerance

• You need external coordination (Consistency)

Page 87: More than Data

SOA on Riak Core

• Riak Core can work for any distributed system

• Riak Core provides:

• Availability

• Partition Tolerance

• You need external coordination (Consistency)

• You provide:

• Management API

• Consistent Responses

• Health Check

Page 88: More than Data

Acknowledgements

• Jordan West (GitHub: jrwest)

• Everyone at StackMob


Data Walls: More than just some DATA on the wall

Azure Mobile Services - more than just cloud data

More than 550,000 data on wheelchair accessibility. Here ...myaccessible.eu/wp-content/uploads/2015/06/MyAccessibleEU-_Ca… · More than 550,000 data on wheelchair accessibility

There is more to Big Data than data

MySQL Security: More Than Just ACL's - Technocationtechnocation.org/files/doc/2012_05_MySQLSecurityPres.pdf · How Does Your Data Flow? ... MySQL Security: More Than Just ACL's

Oracle Active Data Guard Best Practices Standby For More Than … · Oracle Active Data Guard Best Practices Standby For More Than Disaster Recovery Larry M. Carpenter Distinguished

Security and Protection CS 110 Fall 2005. Security Risks More data is being stored than ever before More data is being stored than ever before More people/organizations

Better than Free: Data Explorations with Public Data and ... · Better than Free: Data Explorations with Public Data ... • Pygrib • PyNIO ... • More mainstream usage than NCL

Improve Your Business Performance: Why Meaningful Data is more important than Big Data

Semantic web, linked data and broadcasting - more in common than

MORE THAN YOU IMAGINE Head Teacher Forum Buddy & Financial Data

Information Is More Than Just Data

Is dcom more important than data centre design

Data Warehousing and Bitmap Indexes - More than just some bits

Modern Data Protection Is More Than Backup

Data is More than Knowledge - Oy Meaning Processing · PDF file2 Data is more than knowledge: implications of the reversed knowledge hierarchy for knowledge management and organizational

Do Google Trend data contain more predictability than

More Than Alumni More Than Development More Than …€¦ · More Than Alumni More Than Development More Than What We Do Now. ... has more to do with leadership talent and a drive

Europeana, more than data aggregation?

More than Words - bwgresnet.res.ku.edu · Study 3: Local Sample Qualitative Data Study 1 and 2: National and Local Sample Quantitative Data More than Words: The Relations between

MORE THAN DATA: WORKING WITH BIG DATA FOR …civicdatadesignlab.org/files/big-data-for-civics.pdfI/S: A JOUR MORE THAN DATA: WORKING WITH BIG DATA FOR CIVICS SARAH WILLIAMS According

Big Data: It's More Than Volume, Paypal

Big Data has become a market disruptor in more ways than one. As data … · 2018-12-08 · VP Big Data Kobo Big Data has become a market disruptor in more ways than one. As data

More than a Meal Demonstrating Value- n4a 072317€¦ · More Than a Meal: More Than a Meal: Demonstrating Demonstrating Value Through Enhanced Service Delivery and Outcomes Data

My Cloud is more Secure than your Data Center

More than a Fair Share: Network Data Remanence Attacks

· PDF fileSheraplex finish More than 50 ears More than 50 ears More than 20 ears Stainless Steel More than 50 ears More than 50 ears More than 50 ears Al (Steel) NPD

So Sensitive: Why Data Awareness Matters More Than Ever

High Speed Data Connectivity: More Than Hardware (Design Conference 2013)

Louis Racette Visualizing Data in More Than Three

tout€¦ · tout.ro more than 150.000 sqm more than 100.000 sqm more than 250.000 pcs more than 345 1

Exchanging more than Complete Data

IR is More Than Just Data! - Home - Office of the …provost.tufts.edu/institutionalresearch/files/IUQBPresentation... · IR is More Than Just Data! ... OF PENNSYLVANIA Highest ranked

Learning Analytics: More Than Data-Driven Decisions

Why Market Data Matters Now More Than Ever