intelligent, managed data staging for high-performance computing
TRANSCRIPT
© 2015 IBM Corporation
IBM Platform Data Manager for LSFIntelligent, Managed Data Staging
Gábor Samu, Portfolio Marketing ManagerSoftware Defined [email protected]
© 2015 IBM Corporation2
The IBM Platform LSF Family
IBM Platform
RTM
IBM Platform Analytics
IBM Platform Process Manager
IBM Platform
Application Center
IBM Platform Dynamic Cluster
IBM Platform License
SchedulerIBM
Platform Session
Scheduler
Platform LSF
Operat
ional
and
Manag
emen
t
Repor
ting
Enhancing End
User
Productivity
Scheduling Efficiency
MapReduce Accelerator Platform HPC
DockerConnector
Hadoop Connector
IBM Platform
Data Manager
Platform MPI
Platform PCM
© 2015 IBM Corporation3
Challenges with data management in HPC environments• Data is not available on the compute resources when needed• Transfers of data done “in-band” – idle CPUs waiting for transfers• Wasted bandwidth, storage on duplicate transfers• What is the state of data transfers?• Single user copying the same data repeatedly• Multiple users transferring the same data repeatedly
Compute power requires data to operate on
© 2015 IBM Corporation4
Intelligent, managed data staging
Workload independent, managed data transfers
Eliminate duplicate transfers Lower storage costs Visibility of data transfer traffic
IBM Platform Data Manager for LSF
Whether in the cloud or working locally, ensure that your data is in the right location at the right time with
intelligent caching and out-of-band transfers, helping to reduce costs and overall time to solution.
© 2015 IBM Corporation5
Unique data staging capabilities
• Fully integrated with IBM Platform LSF• Managed movement of data within and between Platform LSF clusters, with control over policies and
priority.
• Control over out-of-band movement of data• Preventing wasted compute cycles.
• Eliminate redundant transfers with intelligent caching of data.• Data affinity
• For environments consisting of multiple IBM Platform LSF clusters, factor in data availability in scheduling
• Configurable file transfer mechanism• Administrators may configure IBM Platform Data Manager to use the underlying file transfer
mechanism (e.g. scp, gridftp, IBM Aspera)
© 2015 IBM Corporation6
Data affinity – a closer look
Cache(file TUV789)
Platform LSF Cluster C Platform Data
Manager
Cache(file XYZ456)
Platform LSF Cluster B Platform Data
Manager
Cache(file XABCD123)
Platform LSF Cluster D Platform Data
Manager
Cache
Platform LSF Cluster A Platform Data
Manager
Platform Data Manager makes data availability a factor when forwarding to remote clusters!
My job requires file XABCD123
Job forwarded to Cluster D, where requested data file is
cached
© 2015 IBM Corporation7
IBM Spectrum Scale (Persistent Data Source)
Flash (Data Manager Cache)
COMPUTE HOSTS
Transfer Job
Platform LSF job asks for
input data file on slow
shared file system
Platform Data
Manager pre-stages the file into fast flash
cache before job execution
Job accesses the data file from
the cache and writes its output to flash
Platform Data
manager drains the
output from the cache
into persistent
storage after job
execution.
HPC perspective – Burst Buffers
Platform LSF&
Platform Data Manager
Transfer Job
© 2015 IBM Corporation8
Thank youFor more information: http://www.ibm.com/systems/platformcomputing/products/lsf