riak intro with azure

&

Thursday, March 7, 13

#WHOISAdron Hall | @adron | Coder, Messenger, Recon

Thursday, March 7, 13

芭蕉

Thursday, March 7, 13

Thursday, March 7, 13

Thursday, March 7, 13

Distributed, masterless, highly-available key/value store

Thursday, March 7, 13

Horizontal Scalability

Fault-Tolerance

Low-latency

Ops Friendliness

Predictability

High-Availability

DESIGN GOALS

Thursday, March 7, 13

When to use Riak...

Thursday, March 7, 13

Metadata Users/Profiles

Object StorageSession Storage

Sensor Data Logging Systems

Record SystemsNotification Systems

RIAK USE CASES

Thursday, March 7, 13

IN PRODUCTION AT

And 1000s more...

Thursday, March 7, 13

DATA MODEL

Thursday, March 7, 13

{“Key”:“Value”}

• Values are stored against keys

• Key/Value + Metadata = Object

• Fundamental Unit of Replication

• Any Datatype will work

• Record to disk in binary format

Thursday, March 7, 13

<<BUCKET>>/<<KEY>>

• Virtual Namespace

• Bucket + Keys = Object Address

• Buckets have properties

• Objects in bucket inherit properties

• No relationships between buckets

Thursday, March 7, 13

DATA ACCESS

Thursday, March 7, 13

INTERFACES

HTTP API - Via a little piece of magic called Webmachine

Protocol Buffers API - Thanks, Google!

Largely-faithful REST implementation

Compact, binary protocol

Thursday, March 7, 13

CLIENT LIBSPython

Ruby

PHP

OCaml

Java

Perl

Erlang

Node.js

C/C++

Haskell

Clojure

Scala

Go Dart

.NET

And more. Supported by either Basho or our community.

Thursday, March 7, 13

RIAK GIVES YOU [FOUR] WAYS TO STORE, RETRIEVE, AND QUERY

DATA

Thursday, March 7, 13

123456789101112131415161718

CRUD

// PUTPUT  /buckets/bucket/keys/key            //  User-­‐defined  key

POST  /buckets/bucket/keys/key        //  Riak-­‐defined  key

DELETE  /buckets/bucket/keys/key      

GET  /buckets/bucket/keys/key        

// GET

// DELETE

Thursday, March 7, 13

MapReduce

Distributed processing system using Riak Pipe

Efficient for targeted queries over known key range

Write jobs in Erlang or JS. (Erlang more performant)

Thursday, March 7, 13

Secondary Indexing (2i)

riak_object

riak_object

X-Riak-Index-email_bin

X-Riak-Index-value_int

“mark@basho.com”

“42”

Tag objects with custom metadata on PUT...Exact match and range queries...No multi-index queries yet...

Pagination is on its way...

Thursday, March 7, 13

Riak Search

Store and index documents (JSON, text, XML, etc)

Current Riak Search supports subset of Solr API

Next iteration (Yokozuna; in beta)will implement distributed Solr on Riak. It will be sexy.

Looking for beta testers to help harden Yokozuna

Thursday, March 7, 13

ARCHITECTURE

The scaleability and ease of operation goals inform architectural decisions.

These come with tradeoffs.

Consistent Hashing

Virtual Nodes

Append-only storage

Handoff/Rebalancing

Vector Clocks

Active Anti-Entropy*

Thursday, March 7, 13

Consistent Hashing

Location of data in the Riak ring is determined based on hash of bucket + key.

Provides even distribution of storage and query load

Trades off advantages gained from locality

- e.g. Range queries and aggregates

Thursday, March 7, 13

Consistent Hashing

Thursday, March 7, 13

Virtual Nodes

Unit of addressing and concurrency in Riak

Each physical host manage many vnodes Partition count / physical machines = vnodes/machine*

Decouples physical assets from data distribution. This provides:

- simplicity in cluster sizing - failure isolation

Thursday, March 7, 13

Handoff/Rebalancing

Mechanisms for data rebalancing

When nodes join/leave cluster, handoff and rebalancing manage the date shuffling dynamically

Trades off speed of convergence vs. effects on cluster performance

- causes disk & network load

Thursday, March 7, 13

Vector Clocks

VCs used to rectify object consistency at READ time.

Lots of knobs to turn; well-documented

Trades off space, speed, and complexity for safety

- will store all sibling objects until resolved- can lead to object size issues

Thursday, March 7, 13

Append-Only Storage

Riak provides a pluggable backend interface. (Write your own; we’ll probably hire you...)

Bitcask, LevelDB are most-heavily used. Both are append - only

Provides crash safety and speed.

Trade off: periodic compaction/merge ops

Thursday, March 7, 13

RIAK 1.3 (AKA “new hotness”)

Active Anti Entropy MapReduce ImprovementsIPv6 Support

Riaknostic included by default

Much more Riak Control improvements

Full release notes: https://github.com/basho/riak/blob/1.3/RELEASE-NOTES.md

Thursday, March 7, 13

FUTURE WORK*(1.4 and beyond)

(* all code subject to ship early, late, or not at all)

Dynamic Ring SizeYokozunaCRDTs/Data TypesRiak Object

Consistency2i ImprovementsRiak Pipe workMuch more

Thursday, March 7, 13

S3-API compatible and supports per-tenant reporting for billing and metering use cases. Additional APIs on the way.

Multi-tenant cloud storage software for public and private clouds.

Designed to provide simple, available, distributed cloud storage at any scale.

Stores files of arbitrary size. Under the hood stores 1MB chunks along side a manifest.

Stateless proxy (CS) does chunking. Riak does distribution, storage, etc.

Thursday, March 7, 13

Data transfer is unidirectional (source -> sink). Bidirectional synchronization can be achieved by configuring a pair of connections between clusters.

Extends Riak's capabilities with:- multi-datacenter replication- SNMP Configuration- JMX-Monitoring- 24x7 support from Basho Engineers

One cluster acts as a "source cluster". The source cluster replicates its data to one or more "sink clusters" using either real-time or full sync.

Thursday, March 7, 13

RIAK COMMUNITYMailing List - 1300 developers

IRC - 200+ people every day yelling about software

GitHub - 1000s of watchers; 200+ contributors to all projects

Meetups - 10 Countries, 23 Cities, 3700+ Members & growing fast!

Deployments - 1000s in production.

Thursday, March 7, 13

May 13-14th in New York Cityricon.io/east.html

Talks, hacking, partiesDedicated to the future of Riak and distributed systems in production

REGISTER NOW!

https://ricon-east-2013.eventbrite.com/?discount=lovevnodes

Thursday, March 7, 13

GETTING STARTEDDownloads - http://docs.basho.com/riak/latest/downloads/Docs - http://docs.basho.comRiak Source Code - github.com/basho/riak

All Basho source Code - github.com/basho/Riak Mailing List - http://bit.ly/riak-list

Email or Tweet me @adron or adron@basho.com

Thursday, March 7, 13

Let’s Talk UI & CLI - Demo Things

Thursday, March 7, 13

#WHOISAdron Hall | @adron | Coder, Messenger, Recon

Thursday, March 7, 13