rakuten leofs - distributed file system
DESCRIPTION
LeoFS has High Performance, High Reliability, High Scalability, file system developed by Rakuten.TRANSCRIPT
1
S3 Compatible Storage “LeoFS”
Rakuten. Inc, RIT Yosuke Hara 25/05/2012
2
Table of Contents
1. Motivation
2. Overview
3. Inside of LeoFS
4. WEB Console
3
1. Motivation
4
Which is suitable storage for storing the media files?
? Low ROI Possibility of SPOF Storage Expansion is difficult during increasing data
Why NFS?
5
6
Object Storage Farm for IaaS
7
2. Overview
8
Storage
Gateway
Manager
Overview
9
LeoFS-Manager
LeoFS-Gateway w/Cache Server
LeoFS-Storage
REST over HTTP RPC
Request from Web Application(s)
META Object Store
Storage Engine/Router
META Object Store META Object Store
RPC
Storage Engine/Router Storage Engine/Router
Load Balancer
S3-API
SNMP
WEB Console
System Layout
10
LeoFS-Manager
LeoFS-Gateway w/Cache Server
LeoFS-Storage
REST over HTTP RPC
Request from Web Application(s)
META Object Store
Storage Engine/Router
META Object Store META Object Store
RPC
Storage Engine/Router Storage Engine/Router
Load Balancer
S3-API
SNMP
GUI Console
Gateway HTTP
Request/Response Handling +
w/Object Cache (AWS S3-API)
Manager Cluster
Management
Ring Watcher Node Watcher
Storage Object Storage, Meta data Storage
+ Replicator/Recoverer
System Layout
11
3. Inside of LeoFS
12
Manager Cluster
Storage Cluster (multi master)
Gateway (stateless)
Process Monitor
Erlang RPC
HTTP
Erlang RPC
Erlang RPC
Architecture
13
LeoFS Gateway REST over HTTP (S3-API)
membership (fault-detection)
redundant-manager replicator
queue
Storage Engine
Object Storage Metadata Storage
LeoFS-Storage
read-repairer
redundant-manager membership (fault-detection)
get put delete head
Architecture - Gateway/Storage
RPC
RPC RPC
Cacher
Architecture
14
Erlang Mnesia
Architecture - Manager Cluster
Gateway / Storage Cluster
RING Member / Cluster State
Auth / ACL
Process Monitor
Architecture
15
Architecture
16
High Cost Performance
Monolithic Storage System
Storage Engine For Unstructured Files
Traffic Restrain Mechanism
> File Cache System (Gateway Plugin)
Three “HIGH”
17
Three “HIGH”
High Reliability
NO SPOF
Split Brain Measure
“Erlang OTP” > Nine Nines (99.9999999%)
18
Three “HIGH”
High Scalability
Elastic Storage System
> Able to dynamic attach/detach nodes
> Able to over 100-nodes
> NOT Mesh-connected Mutual Servers
19
Cache Mechanism
20
High I/O efficiency
Low Latency
Gateway Buffer Pool
Gateway
from Client
Slab Alloc Skiplist
request
{$filename, $etag}
response
match: {ok, match} NOT match: {ok, $metadata, $body,}
Cache Mechanism
21
Storage Engine
22
Metadata + Object Storage
Metadata : Keeps an in-memory index of all data. Object Storage : Log structured (append-only) object store.
Object’s Attribute Storage (metadata) Object Storage
LeoFS-Storage
Storage Engine / Replicator / Recoverer
Storage Engine
23
< META DATA >IDFilenameOffsetSizeChecksum
Metadata
Data
Object Container
Retrieve an object from Storage
Header
File
Footer
< META DATA >IDFilenameOffsetSizeChecksum
Storage Engine
Storage Engine
24
Meta Data Server
Append a File
Insert an object into Storage
Data
Add a Metadata
Storage Engine
Storage Engine
25
Reduce unnecessary files
Compaction
Storage Engine
26
4. WEB Console
27
File Manager
Node Stats
Log Search
WEB Console
Web Console
28
Web Console
Log Analyzer / Searcher
Node State Monitor
29
Storage Gateway Manager
Logstatsh
or
Producers and Admins
GUI Console
Web Console System Layout
Web Console
30
Wrap Up
31
Wrap Up
High Cost Performance High Reliability High Scalability
32
33
Thank you for your time