stanford: solving the hpc data deluge
DESCRIPTION
In this presentation from the Dell booth at SC13, Will Law from Stanford describes how Big Data gets handled for HPC at the University. "As science drives a rapidly growing need for storage, existing environments face increasing pressure to expand capabilities while controlling costs. Many researchers, scientists and engineers find that they are outgrowing their current system, but fear their organizations may be too small to cover the cost and support needed for more storage. Join these experts for a lively discussion on how you can take control and solve the HPC data deluge." Watch the video presentation: http://insidehpc.com/2013/12/03/panel-discussion-solving-hpc-data-deluge/TRANSCRIPT
![Page 1: Stanford: Solving the HPC Data Deluge](https://reader033.vdocuments.site/reader033/viewer/2022052311/55643571d8b42a06138b5028/html5/thumbnails/1.jpg)
HPC Storage at Stanford University!
William Law Stanford Research Computing Center [email protected]!
![Page 2: Stanford: Solving the HPC Data Deluge](https://reader033.vdocuments.site/reader033/viewer/2022052311/55643571d8b42a06138b5028/html5/thumbnails/2.jpg)
We are a new center!
Working across disciplines, across the university.!
![Page 3: Stanford: Solving the HPC Data Deluge](https://reader033.vdocuments.site/reader033/viewer/2022052311/55643571d8b42a06138b5028/html5/thumbnails/3.jpg)
The grand tradition is for each research cluster to be a stand alone island.!
![Page 4: Stanford: Solving the HPC Data Deluge](https://reader033.vdocuments.site/reader033/viewer/2022052311/55643571d8b42a06138b5028/html5/thumbnails/4.jpg)
Backups? Who needs ‘em.! Oh, we do.!
![Page 5: Stanford: Solving the HPC Data Deluge](https://reader033.vdocuments.site/reader033/viewer/2022052311/55643571d8b42a06138b5028/html5/thumbnails/5.jpg)
Storage!
• AFS
• box.com
• GPFS
• Lustre
• NFS
• Full HW solutions (panasas, isilon, netapp)!
![Page 6: Stanford: Solving the HPC Data Deluge](https://reader033.vdocuments.site/reader033/viewer/2022052311/55643571d8b42a06138b5028/html5/thumbnails/6.jpg)
Data Transfer!
• Aspera (Enterprise, Faspex)
• Globus Online
• box.com!
![Page 7: Stanford: Solving the HPC Data Deluge](https://reader033.vdocuments.site/reader033/viewer/2022052311/55643571d8b42a06138b5028/html5/thumbnails/7.jpg)
Concerns!
• IOPS
• Storage Growth
• Sharing/Transfer/Workflows
• Data Integrity
• Backups
• Long Term Data Retention/Sustainability (who pays?)
• Security and Compliance (and dealing with PHI)
• Cost!
![Page 8: Stanford: Solving the HPC Data Deluge](https://reader033.vdocuments.site/reader033/viewer/2022052311/55643571d8b42a06138b5028/html5/thumbnails/8.jpg)
Path Forward!
Standardize platforms Use open protocols for access Provide access to common storage pools across resources!
![Page 9: Stanford: Solving the HPC Data Deluge](https://reader033.vdocuments.site/reader033/viewer/2022052311/55643571d8b42a06138b5028/html5/thumbnails/9.jpg)