osdc 2015: luca gibelli | skylable: storing massive amounts of data, with minimal effort
TRANSCRIPT
Luca Gibelli
www.skylable.com
Today’s presentation
•The limits of traditional SANs
•The limits of the S3 protocol
•Why SX Cluster is better
•Use cases
Find out more: http://www.skylable.com/about
From open-source security …
Find out more: http://www.skylable.com/about
… to open-source storage!
SX Cluster,
open-source
distributed object storage
Seed round by
Sunstone Capital
Find out more: http://www.skylable.com/about
How did it happen?
“We have outgrown
our SAN”
Most pressing issues
“Connectivity between HQ
and satellite offices is poor”
“No vendor offers all
the features we need”
“That new cool project?
Put it on-hold”
“We have
no disaster
recovery plan”
“The new SAN
is over-budget”
Storage:
never big enough
Is that what we need?
1 step back…
The CAP theorem
Block storage: CA
Structured
Limited by metadata
Complex
Expensive
Filesystem storage: CA
Same story
Object-storage: AP
Unstructured
Metadata is in the obj
Application layer
Commodity hardware
Object storage, when…
YOU NEED:
Scalability, durability, low price
FOR:
Media files, documents, backups, log files
Switching is hard
Block/filesystem storage: just syscalls
Object storage: application layer
Must re-engineer the application!
Typicial object storage
makes compromises
PUT, GET, LIST, DELETE
No append
No locking
No file hierarchy (just fakedirs)
What is wrong with S3?
S3
Data Upload Storage
Good enough?
With SX Cluster
you can do better…
SX protocol with
client side
deduplication
Sx
Data Upload Storage
SX is the answer!
Q. No append?
A. Client side deduplication
SX is the answer!
Q. No append?
A. Client side deduplication
Q. No locking?
A. Revisions
SX is the answer!
Q. No append?
A. Client side deduplication
Q. No locking?
A. Revisions
Q. Revisions??
A. Client side deduplication
• Complex to install and run
• S3 protocol only
• No de-duplication
Our competitors
• 3-steps installation process
• SX protocol
• Native de-duplication and
versioning
We do better!
sx@node1 # sxsetup
Enter the cluster name (use the same across all nodes: sx.foo.com
Path to SX storage: /mnt/spare/sx
Maximum size: 1T
Enter the IP address of this node: 192.168.10.1
Is this (192.168.10.1) the first node of a new cluster? (Y/n) <ENTER>
Is this correct? (Y/n) <ENTER>
sx@node1 #
sxsetup on
first node
sx@node2 # sxsetup
Enter the cluster name: sx.foo.com
Path to SX storage /mnt/spare/sx
Maximum size: 1T
Enter the IP address of this node: 192.168.10.2
Is this (192.168.10.2) the first node of a new cluster? (Y/n) n
Please provide the IP address of a working node in 'sx.foo.com'.
IP address: 192.168.10.1
Admin key or path to key-file [default=]:
0DPiKuNIrrVmD8IUCuw1hQxNqZcVPDD82Gkq7PMFYpk3qA8ddxxxxxxx
Is this correct? (Y/n) <ENTER>
Server certificate:
SHA1 fingerprint: 198b0ce5161757ea9bc83fc77627eb8c0958d591
Do you trust this SSL certificate? [y/N] y
sx@node2 #
sxsetup on
additional nodes
The best of 3 worlds
torrent
rsync S3 • Firewall friendly
• Never transfer the same
data twice
• Speed grows linearly
with nr. of nodes
Use cases
What can you do with SX Cluster?
• Drop-in replacement for AWS S3
• Create as many endpoints as you
need and LB between them
• Faster local transfers inside the DC
Use case #1:
S3 compatible storage
• Serve data from multiple locations
• Use S3 to upload/update the data,
serve over http/https
• Replicate synchronously to all
locations
Use case #2:
CDN (credil.org, imgy.org)
• Hundreds of TBs of new medical
images every month
• Plugs into dcm4che
• Multi-tiering
• Offsite live replica, write once, read
many
Use case #3:
Picture Archiving
Communication System
• No middleware!
Scales horizontally
• Multi-platform:
Win/OSX/Linux, Android and
iOS, web
• Client-side encryption
• Incremental transfers,
revisions w/ deduplication
Use case #4:
White label cloud drive
• Commercial support
• SX Console
• Real-time metrics
• Multi-tenancy
• LDAP/AD/SAML + 2FA auth
• SX Indexer
SX Enterprise Edition
Try Skylable
http://www.skylable.com/products/
Build your own distributed object-storage
• GPL licence
• Deploy with a single command
• Scales horizontally
• Parallel transfers, client-side encryption, and
more…
Try Skylable
http://www.skylable.com/products/
SX Cluster
LibreS3
SXDrive
sxfs (beta)
