the kumofs project and messagepack-rpc

The Kumofs Project

and MessagePack-RPCSadayuki Furuhashi

About me

• University of Tsukuba, Japan

• Twitter: @frsyuki_ha

• Blog: http://frsyuki.wordpress.com/

http://frsyuki.wordpress.com

http://frsyuki.wordpress.com

What’s kumofs?

• Distributed key-value store

• “Tokyo Cabinet + Scalability”

• Extreme single node performance (chart)

• Linear scalability of read/write (chart)

• Dynamic rebalance + Consistency control

“key-value store”

• Set(key, value)

• Delete(key)

• value, unique = Get(key)

• CAS(key, unique, value)> if the key is not modified, then set the value.otherwise it fails.

CAS - Compare and Swap

db.atomic do |array| # Get current value. element = array.pop # Modify the value. array # Put the modified value (<--CAS)end # Retry if the value is updated while modifying.

Feature ComparisionData Model Atomic Ops dynamic

rebalance

kumofs

Voldemort

Cassandra

Tokyo Tyrant

Redis

key-value CAS Supported

key-value - Supported

partitioned sorted map table ops Supported

K-V, sorted mapor table

queue ops, etc. -

queue, etc. queue ops, etc. -

Feature ComparisionConsistency Control on

Dynamic Rebalancing

kumofs

Voldemort

Cassandra

Tokyo Tyrant

Redis

on write (original algorithm)

on read (Vector Clock, R+W>N)

on read (Vector Clock, R+W>N)

N/A

N/A

“Tokyo Cabinet + Scalability”

• Extreme single node performance• Dynamic Rebalancing with Consistency Control• CAS support• Linear scalability of read/write

Archtectureof

kumofs

Manager

Gateway

kumo-manager:Manages kumo-servers

Archtecture of kumofs

kumo-server:Replicates and stores data

kumo-gateway:Relays requests from applications to kumo-servers.

Server

Application

ServerManager

Gateway

Manager

Duplicated(HA)

Server

Server

Server

Server

Server

Application

GatewayApplication

Gateway

Replication

Tokyo Cabinet

Server A

Server B

Server C

Server D

Consistent Hashing

hash(ServerA.id)

hash(ServerB.id)

hash(ServerC.id)

hash(ServerD.id)

hash(key1)

Server A

Server B

Server C

Server D

Consistent Hashing

Server A

Server B

Server C

Server D

coordinated by Server B

Server A

Server B

Server C

Server D


Server CServer D

Server A

Server A

Server B

Server C

Server D

set

Replicate

Replication mechanism

Server A

Server B

Server C

Server D

getHigh-availability

Server A

Server C

Server D

getHigh-availability

Technologies of kumofs

• Dynamic rebalancing with Consistency Control- Double-hash-space Algorithm

• Fully multithreaded event-driven I/O- Wavy I/O Archtecture

• Cross-language communication- MessagePack

Server A

Server B

Server C

Server D


Server CServer D

Server A

Adding nodes dynamically

Server A

Server B

Server C

Server D


Server C

Server A

Server E

Move data


Server A

Server B

Server C

Server D

Server E

Data is moving...


Server A

Server B

Server C

Server D

Server E

setget


getset

Consistent Hashing for Reading

Consistent Hashing for Writing

Dynamic rebalacing algorithm of kumofs（double-hash-space）

• Fully multithreaded event-driven I/O

Wainting forevents...

Waiting formutex lock

※ An event arrived ※ More event arrivedThreads

Proceeds event handlersin parallel

Proceeds event handler

The Wavy I/O Archtecture

The Wavy I/O Archtecture

Socket A

Data１

ProceedsData 1

※ Socket-binded I/O (thread-based I/O) ※ Wavy I/O Archtecture

ProceedsData 3

parallel

Proceeds in parallelserializedProceedsData 2

Data２

Data３

Socket B Socket A

Data１

Data２

Data３

Socket B

Cronss-language communication

• Cross-language object serialization library> Management tools := Ruby> kumo-servers := C++> “MessagePack”

• “MessagePack” + “Wavy” is now known as “MessagePack-RPC”

• What about HBASE-794? (New RPC)

What’s MessagePack?

• Efficient serialization library

• Rich data structures - compatible with JSON

• Dynamic typing, Self-describing

• Remote Procedure Call (RPC)

• Synchronous, Asynchronous and Callback style

• Concurrent calls with multiple servers

• Event-driven I/O

• Interface Definition Language (IDL) - compatible with Thrift

Efficient Serialization

1. Compact 2. Fast

･ Zero-copy (C++)

･ Stream deserialization

･ Binary-based format

･ Embed type information

Format of MessagePack

Fixed length types Variable length types

･ Raw bytes

･ Array

･ Map

･ Integer

･ Floating point

･ Boolean

･ Nil

type value type body...length

Type information


JSON MessagePack

null

Integer

Array

String

Map

null c0

10 0a

[20] 91 14

”30” a2 ‘3’ ‘0’

{“40”:null} 81 a2 ‘4’ ‘0’ c0


JSON MessagePack

null

Integer

Array

String

Map

null c0

10 0a

[20] 91 14

”30” a2 ‘3’ ‘0’

{“40”:null} 81 a2 ‘4’ ‘0’ c0

4 bytes 1 byte

2 bytes 1 byte

4 bytes 2 bytes

4 bytes 3 bytes

11 bytes 5 bytes

Type information0x000xc20xc30xca0xcb0xcc0xcd0xce0xcf0xdf...

nil false true float double uint8 uint16 uint32 uint64 int8 ...

Types

0xc00xe0

Type information

Embed value

Positive FixNum

Negative FixNum

FixMapFixArray

FixRaw

0x00

0xc0

0x800x900xa0

0xe0

0x000xc20xc30xca0xcb0xcc0xcd0xce0xcf0xdf...

nil false true float double uint8 uint16 uint32 uint64 int8 ...

Type information Types

Zero-copy serialization

Zero-copy deserialization

Performance

It measured the elapsed time of serializing and deserializing 200,000 target objects. The target object consists of the three integers and 512 bytes string.

Remote Procedure Call

MessagePack-RPC

Inter-process messaging library forclients, servers and cluster applications.

Inter-process messaging library forclients, servers and cluster applications.

Remote Procedure Call

MessagePack-RPC

Concept of Future

Multithreadedevent-driven I/O

Communicates with multiple servers concurrently

Synchronous call

require 'msgpack/rpc'

client = MessagePack::RPC::Client.new(host, port)

result = client.call(:method, arg1, arg2)

Asynchronous callrequire 'msgpack/rpc'


future1 = client.call_async(:methodA, arg1, arg2)future2 = client.call_async(:methodB, arg1, arg2)

result1 = future1.getresult2 = future2.get

Callbackrequire 'msgpack/rpc'


client.callback(:method, arg, arg2) do |future| result = future.getend

client.join

Concurrent calls with multiple servers

require 'msgpack/rpc'

loop = MessagePack::RPC::Loop.new

client1 = MessagePack::RPC::Client.new(host1, port1, loop)client2 = MessagePack::RPC::Client.new(host2, port2, loop)

future1 = client1.call_async(:methodA, arg1, arg2)future2 = client2.call_async(:methodB, arg1, arg2)


Connection Poolingrequire 'msgpack/rpc'

sp = MessagePack::RPC::SessionPool.new

session1 = sp.get_session(host1, port1)session2 = sp.get_session(host2, port2)

future1 = session1.call_async(:methodA, arg1, arg2)future2 = session2.call_async(:methodB, arg1, arg2)



Client

Session Loop Server

Client

ServerSession Loop

sharedevent loop

Client

Client


Server

Server

Loop

Session

Session

Connection Pooling

Session PoolServer

Server

pools these connectionsLoop

Session

Session

Server architecture

Dispatcher

ServerClient

Client

Loop

Event-driven I/O

• Performance of the Loop is important.

• Java version uses Netty (JBoss’ I/O framework)

• Multithreaded

• Utilizes Java’s NIO

• C++ version uses mpio (Kumofs’ I/O architecture)

• Multithreaded

• Utilizes epoll or kqueue

• Ruby version uses Rev (libev for Ruby)

• Utilizes epoll or kqueue

Languages

• Serialization

• C, C++, Java, Python, Ruby, PHP, Perl, JavaScript, D, Lua, Erlang, Haskell, ...

• RPC

• C++, Java, Python, Ruby, PHP, Perl, Haskell, Lua, ...

The MessagePack Project

http://msgpack.sourceforge.net/

The Kumofs Project

http://kumofs.sourceforge.jp/

http://msgpack.sourceforge.net




the kumofs project and messagepack-rpc

Technology