fil og object - ibm · 2017-04-06 · how to build a highly reliable storage system for 1 petabyte...

FilogObjectHvaskalvimeddet?

IBM Systems

IntroducingIBMSpectrumScale

• Removedata-relatedbottleneckswithaparallel,scale-outsolution

• Enableglobalcollaborationwithunifiedstorageandglobalnamespace

• Optimizecostandperformancewithautomateddataplacement

• Ensuredataavailability,integrityandsecuritywitherasurecoding,replication,snapshots,andencryption

Highly scalable high-performance unified storagefor files and objects with integrated analytics

| 2

IBM Systems

Store everywhere. Run anywhere.

• EnterpriseFeatures• Universaldataaccess• Asingleauthenticationscheme• Datadispersalanderasurecodeforfasterrebuildtimes• End-to-endchecksumtocatcherrors• DataprotectionthroughSnapshots,Replication,Backup,and/orDisasterRecovery

• Dataencryptionandcryptographicallysecureerase• IntegrationtoSpectrumFamily

APPLIKASJON

Spectrum ScaleNFS OBJEKTCIFS

Encryption and data governance for compliance

| 3

SpectrumScaleParallelArchitecture

NoHotSpots

• AllNSDserversexporttoallclientsinactive-activemode

• SpectrumScalestripesfilesacrossNSDserversandNSDsinunitsoffile-systemblock-size

• File-systemloadspreadevenly• Easytoscalefile-systemcapacityandperformancewhilekeepingthearchitecturebalancedNSD Client does real-time parallel I/O

to all the NSD servers and storage volumes/NSDs

NSD Client

NSD Servers

Storage Storage

|

4

IBM Systems

SpectrumScaledeploymentmodels

Shared Nothing Cluster (SNC) Model

Span storage rich servers for converged architecture or HDFS deployment

Network Shared Disk (NSD) Model

Modular High-Performance Scaling

Enterprise Integrated Model

Unify and parallelize storage silos

| 5

IBM Systems

AdvantagesofSpectrumScaleRAID

• Useofstandardandinexpensivediskdrives• ErasureCodesoftwareimplementedinSpectrumScale

• Fasterrebuildtimes• Moredisksareinvolvedduringrebuild• Approx.3.5timesfasterthanRAID-5

• Minimalimpactofrebuildonsystemperformance• Rebuildisdonebymanydisks• Rebuildscanbedeferredwithsufficientprotection

• Betterfaulttolerance• Endtoendchecksum• Muchhighermean-time-to-data-loss(MTTDL)

• 8+2P:~200Years• 8+3P:~200MillionYears

JBODs

Elastic Storage Server

Spectrum Scale RAID

| 6

7

• Problem:Separatestoragesystemsforingest/distributionandanalysis• Datamovementoverheadisasignificantpartofmytimetoinsight.• Increasedcostfromdataduplication&overhead• Inconsistentresults

• Solution:NativeHDFSsupport• Decreasedtimetoresults• RunMap/Reducedirectly• Nowaitingfordatatransferbetweenstoragesystems• Immediatelyshareresults

7

Use Case: Big Data Analytics

Spectrum ScaleFile/ ObjectFile/HDFS

GlobalIngestand

DistributionBusinessAnalyticsCustom

Applications

PackagedApplications

IBM Systems

Globalcollaborationoptions

• Singleglobalnamespaceenables:

• RemoteMount• Singlecopyofdata• Usecachingtospeedlocalaccess

• Synchronousreplication• Active/Activedataaccess• Simultaneouswriteissensitivetonetworklatency• Readfromfastestsource• DRwithautomaticfailoverandseamlessfile-systemrecovery

• Asynchronousreplication• Active/Passivedataaccess• Writenow,copylateracrossnetwork• WritetoActive,Readfromfastest• Anystoragetarget,includingcloud

RemoteSite

Primary Cluster

Secondary Cluster

Synchronous replication

Application

Client switchesto secondary

on failure

Whicheveris fastest

Push all updatesasynchronously

| 8

IBM Systems

Store everywhere. Run anywhere.OptimizeCostandPerformance

• Challenge• Datagrowthisoutpacingbudget

• Low-costarchiveisanotherstoragesilo• Flashisunderutilizedbecauseitisn’tshared• Locallyattacheddiskcan’tbeusedwithcentralizedstorage• Migrationoverheadispreventingstorageupgrades

• Automateddataplacement• Spanentirestorageportfolio,includingDAS,withasinglenamespace• Policydrivendataplacement&datamigration• Sharestorage,evenlow-latencyflash• Automaticfailoverandseamlessfile-systemrecovery• LowerTCO

System pool(Flash)

Gold pool(SSD)

Silver pool( NL SAS)

Tape Library

| 9

Dataawarecostoptimization

• Powerfulpolicyengine• InformationLifecycleManagement• Fastmetadata‘scanning’anddatamovement• Automateddatamigrationtobasedonthreshold

• Usersnotaffectedbydatamigration• Singlenamespace

• Example:Onlinestoragereaches90%fullthenmoveall1GBorlargerfilesthatare60daysoldtoofflinetofreeupspace

• IntegratedwithSpectrumArchive

Small files last accessed > 30 days

last accessed > 60days

Silver pool is >60% full Drain it to 20%

accessed today and file size is <1G

Send it back to Silver pool when accessed

System pool(Flash)

Gold pool(SSD)

Silver pool( NL SAS)

Tape Library

Spectrum Archive

Automation

|

10

Dataawareperformanceoptimization

• Alternativetoexplicitpolicies• Respondtochangingworkload

• Dataidentifiedas“Hot”data• High-speedmetadata• Accesspatternanalysis• Migrateclosertoclient

• Flashcanbeaddedanywhere• Readfrom“Fastest”• Latency&cacheaware

Acceleration

System pool(Flash)

Gold pool

Local FlashCache or Tier

|

11

IBM Cloud Object Storage/CleverSafe

EFFICIENCY How to build a highly reliable storage systemfor 1 Petabyte of usable data?

RAID 6 + Replication IBM Cloud Object Storage

1 PB

3.6 PB6003.6x3.6x

3 FTEReplication/backup

Usable Storage

Raw Storage6TB DisksRacks RequiredFloor SpaceOps StaffingExtra Software

$70% +TCO Savings

Original1.20 PB Raw

Onsite mirror1.20 PB Raw

Remote copy1.20 PB Raw

1 PB

1.7 PB2881.7x1.7x.5 FTENone


DIRECT API ACCESS

The Accesser Software exposes three REST APIs for ingest and retrieval. Applications with knowledge of these APIs can leverage IBM COS directly.

Data

Site 1 Site 2 Site 3

S3 Compatible API Openstack Swift Compatible API

Simple Object API

Slicestor Software

Accesser Software


ACCESSMETHODS

DIRECT API ACCESS

The Accesser Software exposes three REST APIs for ingest and retrieval. Applications with knowledge of these APIs can leverage IBM COS directly.

Data



Simple Object API

PARTNER BASED

A variety of certified technology partners can leverage multi tenancy support to satisfy concurrent use cases on a single IBM COS instance.

Slicestor Software

Accesser Software

NAS (CIFS/NFS) Backup/Archive General Applications

Multi Tenancy

Thin ProvisioningSoft/Hard Quotas

HOWITWORKSCONTENT TRANSFORMATION

IBM COS software encrypts, slices and applies Information Dispersal Algorithms otherwise known as erasure coding policies to the data.

Data Ingest

Accesser Software

Storage Nodes


Physical Distribution

Slices are distributed to separate disks and industry standard x86 hardware across geographic locations.

Data Retrieval

Storage Nodes

Reliable Retrieval

An operator defined subset of slices is needed to retrieve data bit perfectly in real time.


BENEFITS

The level of resiliency is fully customizable resulting in a massively reliable and efficient way to store data at scale as opposed to RAID and replication techniques.

Slicestor Software

Accesser Software

SCALABILITY



Simple Object API

Slicestor Software

Accesser Software

Single Nam

espace

Scalability – Scale performance and/or capacity at any time with no downtime to operations

SECURITY

Source:In-Q-Tel

Source:ComputerWorld



Simple Object API

Slicestor Software

Accesser Software

Encryptionfordataatrestprovidesgovernmentgradesecuritywithbuiltinkeymanagement,nosingledisk,nodeorsitecontainsenoughinformationtoconstituteadatabreach.

AVAILABILITY

Nodowntimeduringsoftwareupgrades,hardwarerefreshes,andinthefaceofdisk,node,andsitefailures.



Slicestor Software

Accesser Software

Simple Object API

MANAGEABILITYManagemorepetabyteswithlesspeople. RAID

REPLICATIONSCHEMES

Single Pane of Glass

Comprehensive management via a single web UI

Zero Touch Encryption

No 3rd Party Encryption Solutions Needed

Self Healing

Data is Scrubbed/Corrected 24x7

AronKrishnaStoragedude,IBMSystems

Hotline:[email protected]

fil og object - ibm · 2017-04-06 · how to build a highly reliable storage system for 1 petabyte...

Documents