ibm platform data manager for lsf
TRANSCRIPT
© 2015 IBM Corporation
IBM Platform Data Manager for LSF Intelligent, Managed Data Staging
Gábor Samu, Portfolio Marketing Manager Software Defined Infrastructure [email protected]
© 2015 IBM Corporation 2
The IBM Platform LSF Family
IBM Platform
RTM
IBM Platform Analytics
IBM Platform Process Manager
IBM Platform
Application Center
IBM Platform Dynamic Cluster
IBM Platform License
Scheduler IBM
Platform Session
Scheduler
Platform LSF
Scheduling Efficiency
MapReduce Accelerator Platform HPC
Docker Connector
Hadoop Connector
IBM Platform
Data Manager
Platform MPI
Platform PCM
© 2015 IBM Corporation 3
Challenges with data management in HPC environments • Data is not available on the compute resources when needed • Transfers of data done “in-band” – idle CPUs waiting for transfers • Wasted bandwidth, storage on duplicate transfers • What is the state of data transfers? • Single user copying the same data repeatedly • Multiple users transferring the same data repeatedly
Compute power requires data to operate on
© 2015 IBM Corporation 4
Intelligent, managed data staging
§ Workload independent, managed data transfers
§ Eliminate duplicate transfers § Lower storage costs § Visibility of data transfer traffic
IBM Platform Data Manager for LSF
Whether in the cloud or working locally, ensure that your data is in the right location at the right time with
intelligent caching and out-of-band transfers, helping to reduce costs and overall time to solution.
© 2015 IBM Corporation 5
Unique data staging capabilities
• Fully integrated with IBM Platform LSF • Managed movement of data within and between Platform LSF clusters, with control over policies and
priority.
• Control over out-of-band movement of data • Preventing wasted compute cycles.
• Eliminate redundant transfers with intelligent caching of data. • Data affinity
• For environments consisting of multiple IBM Platform LSF clusters, factor in data availability in scheduling
• Configurable file transfer mechanism • Administrators may configure IBM Platform Data Manager to use the underlying file transfer
mechanism (e.g. scp, gridftp, IBM Aspera)
© 2015 IBM Corporation 6
Data affinity – a closer look
Cache (file TUV789)
Platform LSF Cluster C Platform Data
Manager
Cache (file XYZ456)
Platform LSF Cluster B Platform Data
Manager
Cache (file XABCD123)
Platform LSF Cluster D Platform Data
Manager
Cache
Platform LSF Cluster A Platform Data
Manager
Platform Data Manager makes data availability a factor when forwarding to remote clusters!
My job requires file XABCD123
Job forwarded to Cluster D, where requested data file is
cached
© 2015 IBM Corporation 7
IBM Spectrum Scale (Persistent Data Source)
Flash (Data Manager Cache)
COMPUTE HOSTS
Transfer Job
Platform LSF job asks for input data file on
slow shared file system
Platform Data Manager pre-stages the file into fast flash cache before
job execution
Job accesses the data file from the cache and
writes its output to flash
Platform Data manager drains the output from
the cache into persistent storage after
job execution.
HPC perspective – Burst Buffers
Platform LSF &
Platform Data Manager
Transfer Job