data transfers in the alcf robert scott technical support analyst argonne leadership computing...
TRANSCRIPT
Data Transfers in the ALCFRobert ScottTechnical Support Analyst
Argonne Leadership Computing Facility
Transferring Data To/From ALCF
sftp and scp (for “small” transfers)– If you must use scp, eureka, and in the future tukey are better systems
to scp to/from. All paths will be the same as they are on our other systems.
– Eureka is also a better host for compressing and uncompressing large file archives
GridFTP (for large transfers)– Other site must accept our CA– ssh / cryptocard access available– For more information on Grid FTP please go here:
http://www.alcf.anl.gov/resource-guides/using-gridftp-intrepid Globus Online (for large transfers)
Argonne Leadership Computing Facility - supported by the Office of Science of the U.S. Department of Energy
2
Getting Started with Globus Online
You’ll need a Globus Online account– Go here to signup: https://www.globusonline.org/– Globus Connect: to transfer files to and from your local
machine (laptop)– SSH key or crypto card to access an ALCF endpoint– Go here to initiate your transfer:
https://www.globusonline.org/xfer/StartTransfer
Argonne Leadership Computing Facility - supported by the Office of Science of the U.S. Department of Energy
3
What is Globus Online?
Reliable File Transfer– Easy: Simple signup, login and use – and minimized intervention– Fast: GO can move large filesets in hours– Secure: No Need to worry about security configs or certs, and one-
time passwords work– Research-focused: Globus Online is the only file transfer service with
the scientific researcher in mind No IT required
– No client software installation– Works with existing GridFTP servers– Globus Connect solves “last mile problem” for local machines– Consolidated support and troubleshooting
Argonne Leadership Computing Facility - supported by the Office of Science of the U.S. Department of Energy
4
Transferring Data To/From ALCF using GO Globus Online (for large transfers)
– Globus Online addresses the challenges faced by researchers in moving, sharing, and archiving large volumes of data among distributed sites.
– ALCF BG/P endpoints: alcf#dtn_intrepid, alcf#dtn_surveyor, alcf#dtn
– Check if your laboratory, university or research center has already an endpoint.
– Globus Connect to transfer files to and from your local machine.– http://www.alcf.anl.gov/resource-guides/data-transfer
Argonne Leadership Computing Facility - supported by the Office of Science of the U.S. Department of Energy
5
Globus Online Demo
Let’s start the Demo….
Argonne Leadership Computing Facility - supported by the Office of Science of the U.S. Department of Energy
6
Table of BG/Q File Systems on ALCF Resources
Argonne Leadership Computing Facility – www.alcf.anl.gov
Name System Type Path Production Backed-up Users
vesta-home Vesta GPFS /home Yes No General Use
vesta-scratch(Intensive job output, large files)
Vesta GPFS /projects Yes No I/O
mira-home(general use)
Mira GPFS /home Yes No —
mira-fs0(Intensive job output, large files)
Mira GPFS /projects Yes No I/O
NOTE: all “filesystems” are also available via the /gpfs/$filesystem_name path on these systems.