Intercloud object storage: colony

2012.4.18

Shigetoshi Yokoyama

National Institute of Informatics

@jxta - Trial to use swift

in wide area network -

1

Cloud projects @ NII

・・・

Univ. -A Cloud Univ.-B Cloud

Univ.-X Cloud

NIINII CloudCloud

Academic

Community Cloud Education Cloud Education Cloud

Research Cloud Research Cloud

Science Information Network

Goal: academic community cloud

2

Intercloud services Intercloud services

dodai dodai

Cluster Cluster

IaaS IaaS

VM VM VM VM

Cluster Cluster

IaaS IaaS

VM VM VM VM Virtual

Machines

Physical

Machines

SINET (L2VPN)

Cluster Cluster

IaaS IaaS

VM VM VM VM

Cluster Cluster

IaaS IaaS

VM VM VM VM

colony VMI VMI VMI VMI: Virtual Machine Image

VM: Virtual Machines

Launch machine images

Colony and dodai in a scenario

3

4

Colony

Intercloud object storage service

5

Swift for

intercloud

use

Swift

Swift

Swift

Swift for

intercloud

use

Swift for

intercloud

use

Swift for

local use

Swift for

intercloud use

Nova

Nova

Nova

Glance

Glance Glance

Colony is a project which

federates cloud object

storage services, like swift

to realize intercloud object

storage service.

Swift-I

Cloud-A

Swift-A

Container A1

Container A2

Container A3

Inter-cloud Container I1

Inter-cloud Container I4

Object A1-1

Object A1-2

Object A1-3

Object I4-1

Object I4-2

Object I4-3

Cloud-B

Container B1

Container B2

Container B3

Inter-cloud Container I1

Inter-cloud Container I8

Object B1-1

Object B1-2

Object B1-3

Object I1-1

Object I1-2

Object I1-3

Inter-cloud object storage service : colony

Cloud Services

Inter-cloud Container

I1 Inter-cloud Container I2

Inter-cloud Container I3

Inter-cloud Container I13

Inter-cloud Container I10

Inter-cloud Container I4

Swift-B

Geographically

Distributed Object I4-1

Object I4-2

Object I4-3

Object I1-1

Object I1-2

Object I1-3

Users’ points of view

6

Colony realizes the federation

7

C o l o n y Apache

mod_wsgi mod_shib

Colony-horizon

Colony-keystone

Colony-dispatcher

Squid

Slapd

Ubuntu

Colony-Keystone

Slapd

Swift

Colony-Keystone

Slapd

Swift

Provide seamless access to

multiple swifts

Authenticate with

Shibboleth IdP

Shibboleth IdP

Cloud-A User

Swift-I Swift-A

Developed software components in colony

• Colony-Horizon – based on diablo/stable Horizon with some enhancements

• Multi-region support – Users can choose which swift is used to store/retrieve objects

• Swift Container’s ACL ,metadata support

• Swift Object’s metadata support

• >5G segment upload support …

• Colony-Keystone – based on diablo/stable Keystone with some enhancements

• Authenticate with Shibboleth

• %{tanant_name} can be used for endpointTemplates in addition to %{tenant_id} for

federating cloud services

• Colony-Dispatcher - new

• Relay requests to multiple object services (and merge response for clients)

• Relay requests to a specific object service indicated by URI

• Choose the “nearest” swift-proxy server to relay requests

• Copy objects among different swifts

• Utilities - new

• Tools to make admin task easier to provide the federation of object storage services

8

Swift Swift --AA

Colony-horizon

9

Swift Swift --II Users can choose swift

Shibboleth SPShibboleth SP

10

ColonyColony--HorizonHorizon ColonyColony--HorizonHorizon

Shibboleth IdP

Colony-

Keystone

Colony-

Keystone

Colony – keystone

1. ID/passwd 2. Attribute: ePPN, mail_addr

3. Attribute: ePPN

4. auth_token

0-1. User registration by mail_addr

0-2 . Associate ePPN to mail_addr

by initial access

Modifications to keystone

• Add ePPN field to keystone schema

•

('/ /email')

•

('/users/{

• Add ePPN field to keystone schema

• ADD rest api services to create token by ePPN

('/token_by/eppn') and email address('/token_by/email')

• Add a rest api service to register/update ePPN

('/users/{user_id}/eppn')

Colony-dispatcher

Swift Proxy

Colony Dispatcher

Swift Proxy Swift Proxy

Swift-A (local) Swift-I (intercloud )

A:container1

A:container2

I:container1

I:container2

Swift Client

1. Swift client can send requests to Swift-A and Swift-I through Swift Dispatcher

2. Swift Dispatcher merges the response from each Swift and send it to Swift Client

Requests modified for merging responses.

•Account Info

•Container List

•X-Copy-from/to

Response merged by

Colony Dispatcher has

a prefix to indicate

which Swift is used to

store.

Response merged by

Colony Dispatcher has

a prefix to indicate

which Swift is used to

store.

11

A:container1

A:container2

I:container1

I:container2

Swift Client

Swift Dispatcher can use cache proxy (like squid) per

swift proxy to retrieve objects from remote swifts.

Caching

12

Swift Proxy

Colony Dispatcher

Swift Proxy Swift Proxy

Swift-A (local) Swift-I (intercloud )

Cache(Proxy)

Download performance with cache

13

0

50,000,000

100,000,000

150,000,000

200,000,000

250,000,000

300,000,000

1K(bytes) 1M(bytes) 10M(bytes) 100M(bytes) 1G(bytes) 5G(bytes)

Normal 13,201 11843208 77113446 152263382 163115324 154252275

Cache 382,004 135938185 226800796 256492362 261189612 260956927

byte

s/se

c

Download performance (5times, average)

・AMD Opteron(tm) Processor 6128 2000Mhz 16core ・32GB Memory ・10000baseT 10000Mb/s Full Duplex

Ubuntu Server 11.04 x86_64 Client Colony-Dispatcher Squid Swift-Proxy

・AMD Opteron(tm) Processor 6128 2000Mhz 16core ・32GB Memory ・10000baseT 10000Mb/s Full Duplex

Ubuntu Server 11.04 x86_64 Swift-Storage(account, container, object)x 3

Cache is

effective,

especially for

small size

objects.

Performance : original swift

14

Swift

@Tokyo Swift@ Chiba

Swift @Sapporo

6.15Mbits/sec

6.29Mbits/sec

941.23Mbits/sec

929.15Mbits/sec

7.80Mbits/sec

7.73Mbits/sec

1 2 3 4 5

1K 1,755 2,165 395 3,178 1.375

1M 431,840 446,824 537,768 412,542 43,307

10M 710,179 741,350 737,565 723,446 731,824

100M 780,548 775,131 788,620 782,503 786,224

1 2 3 4 5

1K 11,608 13,293 12,916 14,691 12,126

1M 3,427,827 3,427,402 3,455,788 673,614 3,386,009

10M 958,018 959,098 7,188,901 974,072 949,389

100M 1,109,440 98,400,555 99,641,184 15,411,221 92,400,774

•Upload – Always effected by worst case

•Download – worse case is ramdomly choosed (about 1/3)

Client

Swift cannot perform well when

confgirure it in inter-region.

10G

1G

netperf

Performance : with cache

15

1 2 3 4 5

1K 2,388 2,027 1,629 2,718 2,423

1M 521,456 479,606 488,142 455,359 541,622

10M 727,240 678,511 724,562 730,127 699,518

100M 786,893 777,227 787,707 782,546 784,544

1 2 3 4 5

1K 398,133 415,753 397,979 446,576 406,510

1M 9,238,473 202,193,598 177,694,628 161,992,275 134,450,057

10M 55,186,177 288,776,403 319,892,614 331,691,392 307,500,293

100M 473,135,008 471,484,134 467,654,981 412,153,419 451,464,516

•Upload – Always effected by worst case

•Download (through Cache)

Cache works

Swift

@Tokyo Swift@ Chiba

Swift @Sapporo

6.15Mbits/sec

6.29Mbits/sec

941.23Mbits/sec

929.15Mbits/sec

7.80Mbits/sec

7.73Mbits/sec

Client

10G

1G

netperf

Performance : with zone-awareness

16

1 2 3 4 5

1K 11,356 13,157 13,074 12,758 12,680

1M 9,824,750 11,205,249 7,599,312 10,931,206 11,199,982

10M 52,294,403 51,437,092 51,050,686 52,641,471 52,300,141

100M 97,937,987 101,847,002 102,385,002 102,413,801 101,462,855

•Upload

Modifications for zone-awareness (just a hack) Modifications for zone-awareness (just a hack)

•Uploading • Calculate the number of unfinished tasks in the send queue for each area and when one area are much worth than others, quit the uploading job for its area.

•Downloading • Check connection performance of object servers and try to retrieve an object from the fastest one.

Swift should be zone-aware

for geographically distributed use.

Swift

@Tokyo Swift@ Chiba

Swift @Sapporo

6.15Mbits/sec

6.29Mbits/sec

941.23Mbits/sec

929.15Mbits/sec

7.80Mbits/sec

7.73Mbits/sec

Client

10G

1G

netperf

• Prototyping is done in NII research cloud

https://github.com/nii-cloud/colony/

• Future plan

- Performance enhancements

- Real deployment and evaluation

17

Project status of colony

18

Dodai

Server resource pool

Clusters for each private cloud

Allocate servers

from server resource pool

Release servers

to server resource pool

Private cloud - A Private cloud - B Private cloud - C

CaaS : Cluster as a Service

Elastic

Private

Cloud

19

OpenStack-1 Eucalyptus

Hadoop SunGrid

Engine

Hadoop MPI Hadoop MPI

CaaS

IaaS

PaaS

OpenStack-2

Hadoop MPI

Dynamically and securely separated clusters

…

Upper layer: Deploying software [dodai-deploy]

Lower layer: Preparation of nodes [dodai-compute]

Dodai = dodai-deploy + dodai-compute

20

Dodai-deploy • It is a framework for deployment tool

– New software can be added by anyone

– Now supports

• Openstack Essex

• Openstack Diablo

• Hadoop

• SGE

• It is fast

– Do installation parallel on different machines.

• It is easy to use

– Not just Web UI

– CLI for maintainer

– Rest API for integration

21

How to use dodai-deploy • Install

– Just one command to install all-in-one

• curl https://raw.github.com/nii-cloud/dodai-deploy/master/setup-env/setup-

all-in-one.sh | sudo sh

• Use

– Web UI

• Learn step-by-step guidance at

http://$dodai_deploy_server:3000/

– REST APIs

• Try at http://$dodai_deploy_server:3000/rest_apis/index.html

– CLI

• script/cli.rb list SERVER

22

What’s inside in dodai-deploy

23

Ruby on Rails

MCollective

puppet dodai-deploy server

dodai-deploy nodes

dodai-compute

• In a word

–Bare metal version nova

• Futures

–Provide the same interface as nova to

operate on physical machines

–Separate network segments with Open

Flow Controller

24

What’s inside in dodai-compute

OpenFlow

controller modified nova

OpenFlow

switch

cobbler

25

1. Dodai-deploy

- One of the OpenStack compute automated installations tools http://docs.openstack.org/trunk/openstack-compute/admin/content/openstack-compute-deployment-tool-with-puppet.html

https://github.com/nii-cloud/dodai-deploy/

- Used in NII research cloud development

2. Dodai-compute

https://github.com/nii-cloud/dodai-compute/

- Used in NII research cloud development

3. Future plan

- Enahance based on Feedbacks from NII researchers

- Experiments on intercloud compute service

26

Project status of dodai

27

NII research cloud

gunnii

NII academic cloud : gunnii

dodai-compute

dodai-deploy

Clu

ste

r as a

Serv

ice

on

demand

on

demand

on

demand

28

Deployment Tool

Compute

Based on

bare metal

supported

generalized

“Elastic Private Cloud”

Object storage (Swift)

OpenFlow

Switch/controller

OpenFlow

Switch/controller

NFS

L3 switch

Researchers Researchers

NII shibboleth IdP NII shibboleth IdP

Compute nodes×40 (initial)

GUI server

Deployed Hadoop, OpenStack, Sun Grid Engine

Storage nodes X 8 (initial)

gunnii prototype system

29

Demo ?

30

gunnii is shibboleth ready.

31

Thank you

https://github.com/nii-cloud/