sgi’s lustre hsm - national computational infrastructure · ©2016 sgi sgi’s lustre hsm rob...

2016 SGI2016 SGI

SGIs Lustre HSM

Rob MollardSenior Storage Specialist (APAC)

Some names and brands may be claimed as the property of others

2016 SGI

Why consider Lustre HSM?

Galen Shipman (LANL) Former OpenSFS Chairman

Nov, 2013

Some names and brands may be claimed as the property of others 2

2016 SGI

Data always lives longer than the hardware its stored on.

Forward migration to new technology should never adversely

impact the users.


2016 SGI

Overview of Lustre HSM

Features: Migrate data to and from external storage (HSM) Free disk space when needed Restore data on cache-miss Policy management (migration, purge, soft rm,...) Import from existing backend Disaster recovery (restore Lustre filesystem from backend)


2016 SGI

Supported Lustre HSM Actions

Archive Archiving a file means pre-copying a file from Lustre to an external HSM.

A Copy Tool (copytool) reads file content and copies it to an external HSM.

Once it has been copied, a file is then ready to be released.

Release Remove all file data objects.

Synchronous action which does not involve the copytool nor coordinator.

Restore All file accesses are blocked until the file is fully restored.

Copytool will write file data back from an external HSM to Lustre.

File data accesses are unblocked when the restore is finished.


2016 SGI

Components of Lustre HSM

Coordinator

Policy Engine (robinhood)

Copytool (DMF copytool)

HSM (DMF)

TAPE

ZWS ZWS ZWS ZWS ZWS

TAPE TAPE TAPE TAPE

Tier 2:Disk Cache Manager

Tier 3:Secondary Storage

DMF Managed Environment

copytoolLustre HSM Client AgentPolicyEngine DB

Lustre MDS/MDT

Coordinator

Lustre OSS/OST Building Block


Lust

re C

lient

s


2016 SGI

Lustre HSM | Coordinator Centralises reception of HSM requests and ignores duplicate ones. Dispatches and balances requests across the available copytools. Manages a log of all requests.

Replay request if MDS/MDT has crashed.

Can be manually cancelled

Behaviour is tuneable

Can be stopped, resumed, purged

Retry/No retry on error

Timeouts, number of simultaneous requests, etc

TAPE

ZWS ZWS ZWS ZWS ZWS

TAPE TAPE TAPE TAPE

Tier 2:Disk Cache Manager

Tier 3:Secondary Storage

DMF Managed Environment

copytoolLustre HSM Client AgentPolicyEngine DB

Lustre MDS/MDT

Coordinator


2016 SGI

Lustre HSM | Policy Engine

Policy Engine manages pre-migration and purge policies. A user-space tool which communicates with the MDS and the coordinator.

Watches Lustre file system changes.

Triggers HSM actions/requests like pre-migration, purges and removal.

Lustre HSM Client Agent

PolicyEngine DB

Lustre MDS/MDT

Coordinator



Lust

re C

lient

s


2016 SGI

Policy Engine: Robinhood

Robinhood is a user-space daemon for managing filesystems Purge oldest files when needed

Custom policies

With a database backend Persistent, avoid scanning for each action

Currently MySQL or MariaDB are supported

Supports features like: User/group usage accounting, including file size profiling.

Extra-fast 'du' and 'find' clones.

Customizable alerts on filesystem entries.

Aware of Lustre OSTs and pools. (stripping)

Filesystem disaster recovery tools.

Robinhood User Group 2016 - September 19th, 2016 - Paris, France

DB

MDT ChangeLogs

OST usage

Policies

archive requests(copy data to target)

release requests(free disk space in Lustre)

restore requests(copy data from target)

Robinhood


2016 SGI

Robinhood Policies

Robinhood manages 3 types of policies Migration policy Purge policy Removal policy

Policies - schedule actions on filesystem entries according to admin-defined criteria File class definitions, associated to policies Based on file attributes (path, size, owner, age, xattrs, ...) Rules can be combined with boolean operators Least Recently Used (LRU) based migration/purge policies Entries can be white-listed


2016 SGI

Use casesHPC Active BackupHPC Disaster RecoveryHPC Workflow OptimisationPersistent HPC StorageLower HPC storage costsLong Term HPC ArchivesHPC Active Archive


2016 SGI2016 SGI

Lustre DMF

Active Data Protection


2016 SGI

Data Assurance and ReliabilityA 3-2-1 Approach

3 copies

2

1

med

ia

type

sco

py

offs

ite

Advantage of 3 copies of all data:

Optimized use of storage HW Elimination of backup

Advantage of keeping data on two different media types:

Fast data access Data retention Archive resilience

Advantage of keeping one copy offsite:

Lower power consumption Base for compliance Disaster recovery

1 2 3

Performance copy

Securecopy

Disaster Recovery copy

Primary Data Center

Offsite or Cloud Storage

RAID, Flash, Disc, Tape,

Object & ZWS Tape or Cloud Object

1

1 2


2016 SGI

Lustre DMF Overview

ManagementNetwork

Object StorageTargets (OSTs)

MetadataTarget (MDT)

ManagementTarget (MGT)

Storage servers groupedinto failover pairs

Data Network (LNET)(InfiniBand/Ethernet/Omnipath)

Lustre Clients

Policy EngineServer

DMF Agent(copytool)

StorageMonitoring

Archive

DMF Storage Network(InfiniBand/Fibre Channel/SAS)

Management Server (MGS)Metadata Server (MDS) Object Storage Servers (OSS)


2016 SGI

HSM | Data Migration Facility (DMF)Hierarchical Storage Management

Transparently migrate data to Tape, MAID or Cloud

Lustre*

DMF

Data life cycle management- DMF manages the placement of data

within multiple tiers of storage Automated data migration

- From expensive, production disk to 2nd or 3rd tier storage

Transparent to user- All data appears on line all the time

Key Benefits- DMF reduces tier 1 disk investment- DMF reduces power consumption- DMF protects data long term

SGI DMF 25 years in production Active User Community (DMFUG)

Cloud


2016 SGI

Lustre DMF | Communication & Data FlowLustre* HSM CommunicationsDMF CommunicationsDMF Metadata updateDMF Data

TAPE Parallel DMF Data Mover

Building Block


Building Block

TAPE TAPEParallel DMF Data Mover

Building Block


Building Block

Lustre* HSMClient Agent

PolicyEngine DB

DMF MDS

DMF Managed HSM Environment

Lustre* OSS/OSTBuilding Block

Lustre* OSS/OSTBuilding Block

Lustre* OSS/OST

Building BlockLustre* OSS/OST

Building Block

copytool

Lustre* MDS/MDTBuilding Block

Coordinator

Lustre* Clients Lustre* Clients

Lustre* Clients

Parallel Data Mover Option Data migration from multiple

parallel servers Scales I/O performance Add Additional data movers

as required

ZWS ZWS ZWS

DMF Direct Archiving


2016 SGI

DMF Direct Archiving | Data Flow

DMF Servers

OSS 4

MDS 1 MDS 2

OSS 1

OSS 2

OSS 3

High Performance Disk Cache

Lustre* Clients

Lustre* OSS

Storage

Lustre* MDS

DMF MDS 1

DMF MDS 2

DMF Data Mover

Metadata

Data

Logical PathPhysical Path

PrimaryStorage

TAPE

ZWS DMF Tier2

DMF Tier3


2016 SGI

High performance data migrations DMF Parallel Data Movers (pDMO)

Simplified Lustre HSM communication Single DMF copytool instance

Optimised data migrations DMF Direct Archiving

MAID storage target DMF Zero Watt Storage

Trusted data protection Over 25 years preserving data

Active user community DMF User Group (Feb

2017)http://hpc.csiro.au/users/dmfug/

Key Differentiators


http://hpc.csiro.au/users/dmfug/

2016 SGI

SGI Standardised Lustre Implementation Service Tasks & Deliverables

Project management Pre-sales assessment of storage requirements

- Provide guidance on Lustre best practices- Determine LUN layout- Assess Lustre networking needs

Connect/verify all HW Install Lustre Environment

- Install & configure Lustre SW for servers- Tune Lustre file system environment- Install Lustre clients- Create failover configuration files, if required

Benchmark & Test environment Documentation of system and Lustre networking configuration Knowledge Transfer & Onsite Training


2016 SGI2016 SGI

Lustre DMF

Customer Examples


2016 SGI

Lustre Version: 2.5.41.1 HSM: DMF 6.5 (ISSP 3.5) In use: since early 2015 Capacity: 3PB Performance: ~50GB/s How is it used: Protection of scientific data.

Customer supports users all over France, with Lustre used for persistent storage. Data replication/duplication is not feasible with PBs.


2016 SGI

Lustre Version: 2.5

HSM: DMF 6.5 (ISSP 3.5)

In use: since April 2015

Capacity: 1PB

Performance: ~5GB/s

How is it used: Active Archive (HSM)

Some names and brands may be claimed as the property of others

High PerformanceAutomotive ManufacturingFormula 1 Engine Manufacturer

22

2016 SGI

Lustre Version: 2.5 DDN EXAScaler (Intel EEL) HSM: DMF 6.5 (ISSP 3.5) In use: Testing for several months just going into

production now Capacity: 1.9PB Performance: ~20GB/s How is it used: Robinhood is running in HSM mode,

although there is sufficient capacity for all data to remain online in Lustre (dual state)


2016 SGI

Q & A

24Some names and brands may be claimed as the property of others

SGIs Lustre HSMWhy consider Lustre HSM?Slide Number 3Overview of Lustre HSMSupported Lustre HSM ActionsComponents of Lustre HSMLustre HSM | CoordinatorLustre HSM | Policy EnginePolicy Engine: RobinhoodRobinhood PoliciesUse casesLustre DMFData Assurance and ReliabilityA 3-2-1 ApproachLustre DMF OverviewHSM | Data Migration Facility (DMF)Lustre DMF | Communication & Data FlowDMF Direct Archiving | Data FlowKey DifferentiatorsSGI Standardised Lustre Implementation Service Tasks & DeliverablesLustre DMFSlide Number 21High PerformanceAutomotive ManufacturingSlide Number 23Q & A

sgi’s lustre hsm - national computational infrastructure · ©2016 sgi sgi’s lustre hsm rob...

Documents