a technical overview

28
A Technical Overview Bill Branan DuraCloud Technical Lead

Upload: zeheb

Post on 23-Feb-2016

44 views

Category:

Documents


0 download

DESCRIPTION

A Technical Overview. Bill Branan DuraCloud Technical Lead. Agenda. Introduction Storage Services Reporting UI Security Tools Roadmap. Introduction. DuraCloud is: Hosted service Runs on cloud-based compute systems Connects to cloud-based storage systems - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: A Technical Overview

A Technical Overview

Bill BrananDuraCloud Technical

Lead

Page 2: A Technical Overview

Agenda• Introduction• Storage• Services• Reporting• UI• Security• Tools• Roadmap

Page 3: A Technical Overview

Introduction• DuraCloud is:– Hosted service

• Runs on cloud-based compute systems• Connects to cloud-based storage systems• Provides a service platform

– Open source software suite

• Webinar Focus:– Components of software platform– Interacting with hosted DuraCloud

Page 4: A Technical Overview

File System

DataRepository

User Data Center

DuraCloud Management Console

• Account Management

• Instance Management

DuraCloud in Context

Page 5: A Technical Overview

Administrative User Interface (DurAdmin)

Storage Management (DuraStore)

Service Management (DuraService)

Service Container

Report Management (DuraReport)

DuraCloud Instance

REST APIREST API

REST API

DuraCloud Storage

Page 6: A Technical Overview

Storage Provider Interface

SDSC Cloud Storage

Microsoft Azure

Storage

Rackspace Cloud Files

Amazon S3

User Data Center

Storage Management (DuraStore)

Amazon Storage AdapterREST API

Rackspace Storage Adapter

Azure Storage Adapter

SDSC Storage Adapter

StorageMediation

DuraCloud Storage

Page 7: A Technical Overview

Storage REST InterfaceSpace Actions

• Add Space• Get/Set Space

Properties• Get Spaces List• Get Space Content List• Get/Set Space Access• Delete Space

Content Actions• Add Content• Get/Set Content

Properties• Get Content• Copy Content• Delete Content

Other Actions• Get Stores• Get Tasks List• Perform Task

Storage Provider Interface

Page 8: A Technical Overview

Storage Adapters

Transform calls from DuraCloud interface into calls understood by a specific storage provider• Determine how DuraCloud “spaces” translate

into top level provider containers (buckets)• Define how space and content properties are

stored and retrieved• Manage eventual consistency concerns

Page 9: A Technical Overview

Administrative User Interface (DurAdmin)

Storage Management (DuraStore)

Service Management (DuraService)

Service Container

Report Management (DuraReport)

DuraCloud Instance

REST APIREST API

REST API

DuraCloud Services

Page 10: A Technical Overview

DuraCloud Services

Bit Integrity Checker

Bit Integrity Checker

Tools

Bit Integrity Checker

Bulk

Duplicate on

Demand

Duplicate on

Change

Media Streame

r

Image Transform

er Bulk

Image Transform

er

Image Server

… so how are they deployed?

Page 11: A Technical Overview

Service Manager

REST API

Service Management (DuraService)

Service Registry

Service Bundle

Service Config

Service Plan

Services Availabl

e?

Services In Plan?

Service List

12

3

4

Service List

Part 1: Ask which services are available

DuraCloud Service Deployment

Page 12: A Technical Overview

Service Manager

REST API

Service Management (DuraService)

Service Registry

Service Bundle

Service Config

Service Plan

DuraCloud Service Deployment

DeployService

Retrieve Service Bundle

Service Bundle

12

3

Part 2: Deploy Service

Service

Config

Service Configurator

Deployed Service Config

Deployed Service

Service Container

4Deploy ServiceServic

e Config

Service

Bundle

Service

Bundle

Page 13: A Technical Overview

Service REST Interface

• Get Services• Deploy Service• Get (Deployed) Service• Get Deployed Service

Properties• Update Service Configuration• UnDeploy Service

Page 14: A Technical Overview

DuraCloud Instance Services

Bit Integrity Checker

Bit Integrity Checker

Tools

Bit Integrity Checker

Bulk

Duplicate on

Demand

Duplicate on

Change

Media Streame

r

Image Transform

er Bulk

Image Transform

er

Image Server

DuraCloud Distributed

Services

• Runs on DuraCloud Instance• Connects to DuraStore• Can be direct Java service• Can be deployed web app

• Runs primarily outside of DuraCloud Instance

• Connects to DuraStore• Makes use of cloud network

or computation features

Page 15: A Technical Overview

Service Manager

REST API

Service Management (DuraService)

Service RegistryBundle

Config

Service Plan

DuraCloud Bulk Service Deployment

1

2

3

ConfigDeploye

d Service

Service Container

4

Task Management (DuraStore)

REST API

Hadoop

5

Run Service

Task6

Run Hadoop

Job

Page 16: A Technical Overview

Administrative User Interface (DurAdmin)

Storage Management (DuraStore)

Service Management (DuraService)

Service Container

Report Management (DuraReport)

DuraCloud Instance

REST APIREST API

REST API

DuraCloud Reporting

Page 17: A Technical Overview

Report Management(DuraReport)

Storage Management (DuraStore)

Service Management (DuraService)

REST API

DuraCloud Report Generation

Message

Broker

Storage Report Builder

Service Report Builder

Deposit Reports

Service Event Messages

REST API

Storage Data

Page 18: A Technical Overview

Report REST InterfaceStorage Report

• Start Storage Report• Get Latest Storage Report• Get Storage Report List• Get Storage Report• Get Storage Report Info• Cancel Storage Report• Schedule Storage Report• Cancel Storage Report

Schedule

Service Report• Get Deployed Services

Report• Get Latest Completed

Services Report• Get Completed Services

Report List• Get Services Report

Page 19: A Technical Overview

Administrative User Interface (DurAdmin)

Storage Management (DuraStore)

Service Management (DuraService)

Service Container

Report Management (DuraReport)

DuraCloud Instance

REST APIREST API

REST API

DuraCloud UI

Page 20: A Technical Overview

Storage Management (DuraStore)

Service Management (DuraService)

REST API REST APIReport

Management (DuraReport)

REST API

Administrative User Interface (DurAdmin)Store Client Service

Client Report Client

DuraCloud UI Interaction

Page 21: A Technical Overview

DuraCloud SecurityStorag

e Provide

r

DurAdmin

DuraStore DuraService

DuraReport

Application SecurityStorage Provider Access Control

User Data Center

Transport Security

Instance Firewall

Page 22: A Technical Overview

DuraCloud ToolsTransfer to DuraCloud

• Upload Tool– GUI to select files and

folders– Visual transfer progress

• Sync Tool– Monitors for changes– Run one time or

continually• Chunk Tool

– Transfer large files– Acts on a single file

Transfer from DuraCloud• Retrieval Tool

– Retrieve files from one, selected, or all spaces

• Stitch Tool– Retrieve large files– Acts on a single file

Page 23: A Technical Overview

Storage Management (DuraStore)

REST API

DuraCloud Tools

User Data Center

Store ClientChunk Tool

Sync Tool

Store Client

Stitch ToolRetrieval

ToolUpload Tool

Page 24: A Technical Overview

Near Term Road MapNew features• Access control at the space level• Improved format identification• Automated bit integrity checking• Email notification on service completion• Multi-Account DuraCloud instances• Content indexing and Search

– Improved discovery– Improved content browsing

• Folder-based navigation• Improved handling of very large files• Multi-Item updates/deletes

Integrations• SDSC storage provider (production version)• Fedora-in-the-cloud prototype• DSpace-in-the-cloud prototype

Page 25: A Technical Overview

Long Term Road MapNew Features • Video transcoding• Document format transformation• Improved image support services• Media streaming access controls• Metadata extraction services Integrations• Azure storage provider (production

version)• Eucalyptus Walrus storage provider• Rackspace compute instances• Managed Fedora-in-the-cloud• Managed DSpace-in-the-cloud

Page 26: A Technical Overview

Questions?

Page 27: A Technical Overview

Upcoming DuraCloud Webinars

Fedora and DuraCloudJanuary 11, 2012 at 1pm ET

DSpace and DuraCloudNovember 30, 2011 at 1pm ET

Page 28: A Technical Overview

Thank You!

Further Questions?

[email protected]