duracloud integration with sdsc presentation slides

SDSC cloud storage integration with DuraCloud

Michele Kimpton, CEO DuraSpaceMatthew Kullberg , SDSC Technical Projects Mgr

Carissa Smith, Partner Specialist DuraSpace

Using the Webinar Platform

• 2-way audio for all participants is muted

• We’ll utilize the Chat Window for the Q&A portion or you may use it if you are having technical difficulties

• You may type your question here & hit ‘enter’

A not for profit serving academia

We are committed to providing open source technologies and services that promote durable, persistent access to

the scholarly record.

What is DuraCloud?Archiving and preservation services in the cloud

Ability to choose one or multiple cloud storage providers

Amazon SDSC Rackspace

Key servicesDigital archiving Video streaming

Image serving

Online sharing

File health checking File synchronization

Why we are excited about SDSC partnership

• First academic public cloud provider• Understands our community• Level of transparency• Cost competitiveness• Real partnership

DuraCloud Pricing

Cheaper than storing content at Amazon

Institutions currently archiving in two data centers using DuraCloud

• Rice• MIT• ICPSR• NC State Archives and Libraries• University of Tennessee Knoxville

Academic Cloud digital archiving network

DuracloudDuracloud

DuracloudDuracloud

DuracloudDuracloud

DuracloudDuracloud

DuracloudDuracloud

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

11/8/2012

Matthew KullbergTechnical Project and Services Manager

San Diego Supercomputer Center

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

A Paradigm Shift for Long-Term Storage: Access, Sharing and Collaboration

SDSC Cloud •Launched September 2011•Largest, highest-performanceknown academic cloud at launch•5.5 Petabytes initial capacity•8 GB/sec throughput•Capacity and performance scale linearly to 100’s of petabytes•Open source platform based on NASA and RackSpace software•http://cloud.sdsc.edu

11

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Why OpenStack Swift Cloud Software?

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

SDSC Cloud Storage design using OpenStack

Authentication nodes provide user access

Proxy nodes handle data read/write requests

Storage nodes write and verify 2 copies of alldata uploaded to the cloud

Authentication nodes provide user access

Proxy nodes handle data read/write requests

Storage nodes write and verify 2 copies of alldata uploaded to the cloud

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Where is the SDSC Cloud?• The San Diego Supercomputer Center is an Organized Research

Unit of the University of California, San Diego

• SDSC Cloud infrastructure is housed within the SDSC Datacenter• Multiple 10Gb network connections including Internet2 and Esnet • Operations staff on site 24/7 • Datacenter is secured with biometric access and 60 closed circuit cameras • Central UPS provides 3.5 Megawatts of battery backup power to the SDSC

Cloud infrastructure and Datacenter• All Hardware resides on seismic isolation platforms from ISOBase to

protect equipment during earthquakes

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

What does my data do while at SDSC?• All Objects (files) are continually checked for errors to ensure

two identical copies exist at all times within SDSC’s Cloud • If an error is detected, the “bad” file is quarantined and a new duplicate

copy is made from the “good” copy.• On average, with enterprise hard drives, data corruption occurs every 1015

bit written, or every 1,000,000,000,000,000 bit written. (Known as UBE, Unrecoverable Bit Error Rate).

• Data remains private at all times, unless shared by the user• SDSC will never mine stored data for any research, commercial, or other

purpose.• All SDSC ITS Staff undergo background, World Check, Consumer, and Live

Scan verifications, and have Level-C clearance.

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Why add SDSC Cloud to your data profile?

• Increased redundancy• SDSC and DuraCloud understand the needs of

academic institutions• Low cost, High Availability and High Durability• Easily move or mirror data with DuraCloud• Recognized by NSF and other academic funding

agencies• Help support academic computing and research

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Our Academic Storage Mission: Data Curation, Archiving, Preservation

• Data Curation: Managing data to ensure they are fit for contemporary use and available for discovery and reuse.

• Archiving: Ensuring that data are properly selected, appraised, stored, and made accessible. The logical and physical integrity—including security and authenticity—of the data are maintained.

• Preservation: Ensuring that items or collections remain accessible and viable in subsequent technology environments.

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Applications of SDSC Cloud

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

What’s on the horizon?• SDSC OpenStack Compute Environment in development

• Allows SDSC to act as primary storage option with DuraCloud• SDSC North Cloud now in testing phase

• Northern Datacenter is in Oakland, CA at the University of California Office of the President (UCOP) Datacenter

• Potential for SDSC North Cloud to provide additional geographically dispersed storage locations through DuraCloud

• SSD Drives currently being installed in SDSC Cloud nodes• Moves database activities to low latency and high I/O Solid State Drives• Provides faster authentication• Improves file and container access speeds• Faster uploading

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Hope to see you in the cloud soon!If you’d like to know more about the SDSC Cloud

infrastructure please email me at: mck@sdsc.edu

To sign up with DuraCloud please visit:www.DuraCloud.org

Matthew C. KullbergTechnical Project and Services Manager

San Diego Supercomputer Center

Time for a quick demo…

To find out more

• www.duracloud.org• wiki.duraspace/display/duracloud• www.youtube.com/user/duracloudvideos• Email us a mkimpton@duraspace.org, or

csmith@duraspace.org

To find out morewww.duracloud.org