Download - Ultra Users Guide v1.2
-
7/29/2019 Ultra Users Guide v1.2
1/35
HPCC USERS GUIDEVersion 1.2
July 2012
IITS (Research Support)Singapore Management University
-
7/29/2019 Ultra Users Guide v1.2
2/35
Revision History
Version 1.0 (27 June 2012):- Modified HPCC users guide after creating the new Ultra HPC
clusterVersion 1.1 (3 July 2012):
- Modified HPCC users guide after changing the job queueselection option in the HPC job submission script
Version 1.2 (25 July 2012):
- Modified HPCC users guide after adding new job queue selectionoption in the HPC job submission script
-
7/29/2019 Ultra Users Guide v1.2
3/35
Table of Contents
Introduction ..................................................................................... 4
HARDWARE ....................................................................................................... 4
SOFTWARE ....................................................................................................... 4HOW TO USE THIS GUIDE ................................................................................... 5
Chapter 1:Accessing HPC Cluster ....................................................... 6
1.1. Download Secure Shell Program ................................................................. 6
1.2. Connect to Ultra on SSH ............................................................................ 61.3. File Transfers to / from your Home Directory on Ultra .................................... 7
Chapter 2:Job Scheduler: Univa Grid Engine ....................................... 8
2.1. Grid Engine Commands ............................................................................. 8
2.2 Job Logs: Error and Output Streams .......................................................... 10
Chapter 3:Software Job Submission ................................................. 11
3.1. Overview ............................................................................................... 11
3.2. MATLAB ................................................................................................. 163.3. MATLAB Compiler .................................................................................... 173.4. GAUSS .................................................................................................. 183.5. R .......................................................................................................... 20
3 6 STATA SE 21
-
7/29/2019 Ultra Users Guide v1.2
4/35
Introduction
HARDWARE
The High Performance Computing (HPC) Cluster is available in SMU for researchers needing
computational resources to run numerical analysis and simulations on the Linux plat form. Managed
by the Integrated IT Services (IITS) Research Support, the cluster comprises of:-
a. 1 x Intel Xeon CPU X5460 3.16GHz 8 cores processors compute server with 16GB acting as
the frontend login host
b. 8 x Intel Xeon CPU X5460 3.16GHz 8 cores processors compute server with 32GB RAM per
server
c. 2 x Intel Xeon CPU E5620 2.40GHz 80 cores processors compute server with 512GB RAM
per server
d. 2 x Intel Xeon CPU X7542 2.67GHz 24 cores processors compute server with 128GB RAM
per server
The servers are connected with a 10 Gigabit Ethernet switch on a private network for high speed low
-
7/29/2019 Ultra Users Guide v1.2
5/35
HOW TO USE THIS GUIDE
This guide details the information required for you to use the HPC cluster.
Chapter 1 (Accessing HPC Cluster) tells you how you can connect to the cluster from inside and
outside of the SMU network.
Chapter 2 (Job Scheduler: Univa Grid Engine) touches on the three Linux commands that you need
to know to run and manage jobs in the cluster.
Chapter 3 (Software Job Submission) provides you with information required to run jobs for the
specific software in the cluster.
The appendixes provide quick referrals or refreshers when youre using the cluster:-
Appendix A (Linux Commands for home directory administration) shows some Linux commands for
self administration of your home folder.
-
7/29/2019 Ultra Users Guide v1.2
6/35
Chapter 1.
Accessing HPC Cluster
Please apply for a HPC cluster account with IITS via email ([email protected]) if you do
not have an account. Otherwise you risk locking up the password to your SMU account if you
follow the steps shown in this guide.
The Ultra cluster is a shared resource. The current user disk quota is set at 12GB (soft limit) and
15GB (hard limit). Please ensure that you have sufficient disk space before running large data jobs.
If you need additional disk space, please email us [email protected].
To login to the cluster, you will need a secure shell program or any Linux-based OS. Suggestedsecure shell programs include:-
1.1. Secure Shell Program
1. Putty
http://www.google.com.sg/search?hl=en&q=putty&meta=
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]://www.google.com.sg/search?hl=en&q=putty&metahttp://www.google.com.sg/search?hl=en&q=putty&metamailto:[email protected]:[email protected] -
7/29/2019 Ultra Users Guide v1.2
7/35
1.2.1 Using Putty.exe
Save the file putty.exe to your computer and double-click to run it. You will see the interface below:-
Figure 1.1: Putty Settings
-
7/29/2019 Ultra Users Guide v1.2
8/35
Figure 1.3: WinSCP Settings
Fill in the hostname, user name, password and choose the SFTP protocol as detailed in Figure 1.3
above. Once logged in, you can proceed to transfer files between the 2 displayed windows (one of the
windows will be your desktop while the other will be your home directory in Ultra).
Chapter 2.
Job Scheduler: Univa Grid Engine
-
7/29/2019 Ultra Users Guide v1.2
9/35
This command is used to submit a job description file (which describes the job you wish to run) for job
queuing and execution. More information on specific software job description file will be provided in
the later chapter which discusses the software in the cluster.
Syntax:
qsub [ job script ]
[ job script ] Job description file to submit
Example:
To submit a job: qsub jobscript.sh
2.1.2. qstat (to query job status)
This command is used to query the status of submitted jobs and queues. Use this command after you
submit a job to check whether the job is running.
Syntax:
qstat [ -f ] [ -u userid ] [ -s {r|p|s|z} ] [ -help ]
[ -f ] Optional. Display the full list of compute nodes and jobs running or queued
[ -u userid ] Optional. Display jobs belonging to the userid
[ -s {r|p|s|z} ] Optional. Display jobs in the specified state of r (running), p (pending), s(suspended) orz (zombie). Can be used in combination, example -s rpsz.
[ h l ] O ti l Di l i k h l
-
7/29/2019 Ultra Users Guide v1.2
10/35
2.2 Job Logs: Error and Output Streams
For every job submitted, the screen output and errors will be dumped into a file named after your job
submission script in the following format:
output log: job_name.sh.ojob_id
If the job did not run as expected and encountered errors, the file might log some information to aid
troubleshooting.
-
7/29/2019 Ultra Users Guide v1.2
11/35
Chapter 3.Software Job Submission
The following are some of the computational codes that can be run in the Ultra cluster:
1. MATLAB codes
2. MATLAB-compiled binaries
3. Gauss-compiled pseudo-code
4. R
5. STATA SE
6. ILOG CPLEX
7. C/C++ and FORTRAN codes (uses the Intel, Pathscale or GNU compiler)
8. Java Codes
Code samples and instructions to run the codes are available in users home directory under the
subfolder ultra-samples.Please con tact IITS (email:[email protected])for assistance for
sof tware not l is ted above.
mailto:[email protected]:[email protected]:[email protected]:[email protected] -
7/29/2019 Ultra Users Guide v1.2
12/35
You need to create a job submission script for every job that you intend to submit. This job submission
script contains the Univa Grid Engine switches (parameters) and software specific instructions to run
your code. This section shows you how to write a job submission script.
A job submission script consists of 2 parts:
1. Univa Grid Engine (GE) switches
2. Software specific instruction
Example of a job submission script:
#! bin/bash
#$ -j y#$ -cwd#$ -m e#$ -M [email protected]
#$ -q short.q
./sim1
Grid Engine switches:
GE switches
Software specifics
-
7/29/2019 Ultra Users Guide v1.2
13/35
Software Launching Instructions
C / C++ / FORTRANcompiled binary
./[compiled filename] > [ output file ]
example:
./sim1 > out.txt
Gauss pseudocode gsrun b [ gauss filename ]
example:
gsrun -b ols.e.gcg
ILOG CPLEX cplex < [ cplex script ] > [ output file ]
example:
cplex < sim1 > result.txt
JAVA . /etc/profile.d/java.sh
java [ java classname ]
example:
. /etc/profile.d/java.sh
java testrun > results.txt
MATLAB matlab -r '[ filename ]'
example (the.m f i le extension m ust be om it ted):
-
7/29/2019 Ultra Users Guide v1.2
14/35
1. Name the script with a file extension .sh to indicate that the file is a shell script
2. Start the script with the line: #! /bin/bash
3. Add the Grid Engine switches
4. Add the software specific instructions
The following is an example of a job submission script named mysim1.sh which is used to run a 64-
bit MATLAB compiled binary file named sim1:
Example job submission script sendjob1.sh:
#! /bin/bash
#$ -j y
#$ -cwd#$ -m e
#$ -q short.q
mailto:[email protected]:[email protected]:[email protected] -
7/29/2019 Ultra Users Guide v1.2
15/35
Example output:
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
110 0.55500 sendjob1f. kenyamada r 06/28/2012 18:57:50 express.q@c3-1 1
123 0.55500 sendjob1f. kenyamada r 06/28/2012 19:44:20 express.q@c3-2 1
126 0.55500 sendjob1f. kenyamada r 06/28/2012 19:44:35 express.q@c3-2 1
Possible states are:-
- r: running
- qw: queuing
- t: transfer
- Eqw: error queuing
Alternatively, run the command jobwatch which displays per second update of qstat. To break out
of jobwatch, press Ctrl-C.
Note: If your job state is Eqw, you need to delete the job. Then check the error and output file for
hints on why the job could not run. You may also want to double-check your job-script for typo
mistakes. Then you can resubmit the job again.
-
7/29/2019 Ultra Users Guide v1.2
16/35
3.2. MATLAB
You can run MATLAB codes directly in the cluster. However, you are encouraged to compile MATLAB
codes instead as compiled MATLAB jobs do not utilize network licenses. More information on
MATLAB compilation can be found in the next section.
3.2.1 MATLAB Codes
MATLAB codes are interpreted by the software and are platform independent. In most cases,
MATLAB codes can be ported between Windows and Linux platforms. However, note that there is a
file path difference between the codes on Windows and Linux. On Windows, file paths are written like:
D:\run1\compute.m
while on the Linux platform, it is written as:
/home/[ your userid ]/run1/compute.m
I f you use such f i le path references in you r code, you have to change i t accordingly w hen you
port the codes over to the cluster .
-
7/29/2019 Ultra Users Guide v1.2
17/35
3.3. MATLAB Compiler
The MATLAB Compiler allows MATLAB codes to be compiled into binary (executable) form that can
be run across the same platform using the MATLAB Run-Time Library. Running a MATLAB
executable does not require any MATLAB network licenses.
To compile MATLAB codes, scripts must be first be marked as functions. This usually involves putting
in a header line like function main. This command then compiles the code inside MATLAB:
>> mcc mv simulation1.m
Using the above example, a few related files will be created in the folder:
1. simulation (this is the compiled binary)
2. files with extension .c (these can be removed)
3. simulation.prj (this can be removed)
4. simulation.log (this can be removed)
5. run_simulation.sh (this can be removed)
3.3.1. Compiled MATLAB Job Submission Script
-
7/29/2019 Ultra Users Guide v1.2
18/35
3.4. GAUSS
There are no GAUSS licenses available on the compute nodes. Instead, the compute nodes are
installed with the GAUSS Run-Time Module that can run GAUSS-compiled pseudo-codes. On the
cluster, the front-end node has the following GAUSS packages:
1. Constrained Maximum Likelihood add-on
2. Constrained Optimization add-on
3. GAUSS Run-Time module
3.4.1 Gauss Codes
GAUSS codes are interpreted by the software and are platform independent. In most cases, GAUSS
codes can be ported between Windows and Linux platforms. However, note that there is a file path
difference between the codes on Windows and Linux. On Windows, file paths are written like:
D:\run1\compute.m
while on the Linux platform, it is written as:
-
7/29/2019 Ultra Users Guide v1.2
19/35
In the following example, compute.e.gcg is the GAUSS pseudo-code file that we want to run. We
create a job submission script sendjob1.sh to describe compute.e.gcg as the Gauss pseudo-code
file to be run:
Example job submission script sendjob1.sh:
#! /bin/bash
#$ -j y
#$ -cwd
#$ -m e
#$ -q short.q
gsrun -b compute.e.gcg
The following command submits the job submission script:
qsub sendjob1.sh
mailto:[email protected]:[email protected]:[email protected] -
7/29/2019 Ultra Users Guide v1.2
20/35
3.5. R
R is available on all nodes in the cluster.
3.5.1. R Job Submission Script
Just like Matlab and Gauss, a job submission script has to be created in order to submit a job. In this
example, simulation.R is the R code that needs to be submitted. A job submission script
sendjob1.sh can be created to point simulation.R as an R code to be run:
Example job submission script sendjob1.sh:
#! /bin/bash
#$ -j y#$ -cwd
#$ -m e
#$ -M [email protected]
#$ -q short.q
R -q --vanilla < simulation.R
-
7/29/2019 Ultra Users Guide v1.2
21/35
3.6. STATA SE
STATA SE is available only on the Intel EM64T and AMD64 nodes in the HPC Cluster.
3.6.1. STATA Job Submission Script
Just like all other jobs, a job submission script has to be created first in order to submit a STATA job.
In this example, sortdata.do is the STATA code that needs to be submitted and run. A job
submission script sendjob1.sh can be created to describe sortdata.do as the code to be run:
Example job submission script sendjob1.sh:
#! /bin/bash
#$ -j y
#$ -cwd
#$ -m e
#$ -q short.q
mailto:[email protected]:[email protected]:[email protected] -
7/29/2019 Ultra Users Guide v1.2
22/35
3.7. ILOG CPLEX
ILOG CPLEX is available only on the 64-bit nodes in the HPC Cluster.
3.7.1. ILOG CPLEX Job Submission Script
Submitting an ILOG CPLEX job is slightly different from the other software in the HPC Cluster. There
are 3 components instead of the usual 2 components (software job and submission script). Submitting
an ILOG CPLEX job requires 3 files:
a. CPLEX problem formulation
b. Script file containing CPLEX commands
c. Cluster submission script containing submission characteristics of job
The following illustrates an example (also found in the sub-folder Ultra-samples/ilog) for submitting a
CPLEX job:
(a) create the CPLEX problem:
(example file name problem.lp)
------------------------------
maximize
x1 + 2 x2 + 3 x3
-
7/29/2019 Ultra Users Guide v1.2
23/35
(example file name cplex-submit.sh)
------------------------------
#! /bin/bash
#$ -j y
#$ -cwd
#$ -m e
#$ -M [email protected]
#$ -q short.q
cplex < cplex-script
------------------------------
The following command submits the job submission script:
qsub cplex-submit.sh
-
7/29/2019 Ultra Users Guide v1.2
24/35
3.8. Intel C / C++ / FORTRAN
The cluster front-end node is installed with the Intel Compiler Suite (C/C++ and FORTRAN compilers,
Math Kernel Library, Performance Primitives and Vtune Analyzer) as well as the GNU compilers. The
compute nodes only have the Intel run-time libraries and the GNU compilers.
The Intel compilers have been optimized for the Intel platform (here 64-bit architecture) while the GNU
compilers are generalized x86 open-source compilers. For this cluster, it is recommended that you
use the Intel compilers unless you prefer the GNU compilers.
3.8.1 C / C++ Compiler
To run C/C++ codes on the 32-bit platform of the cluster, the codes must first be compiled into binary
form. To compile a file:
icc static [ source file name ] -o [ output file name ]
If the code is compiled successfully, it can then be dispatched for running by creating and submitting a
job submission file. See section below.
3.8.2. FORTRAN Compiler
-
7/29/2019 Ultra Users Guide v1.2
25/35
#$ -cwd
#$ -m e
#$ -q short.q
./run_c_code
The following command submits the job submission script:
qsub sendjob1.sh
mailto:[email protected]:[email protected]:[email protected] -
7/29/2019 Ultra Users Guide v1.2
26/35
3.9. IMSL Numerical Library (32-bit )
The IMSL Numerical Library is a 32-bit numerical library and is available on the 64-bit nodes.
3.9.1. IMSL FORTRAN Job Submission Script
The example below is taken from the sub-folder Ultra-samples/imsl in the home directory ofall user.
The file README in the sub-folder also explains how you can compile and run the job. In this
example, lin_eig_self_ex4.f90 is the FORTRAN code that uses IMSL functions and needs to be
compiled. A job submission script called imsl-submit.sh can be created to include lin_eig_self_ex4
as the FORTRAN code to be run:
2 methods are suggested below to compile the FORTRAN code that uses IMSL functions:
(a) First, setup the environment for the IMSL libraries:
. /opt/vni/CTT6.0/ctt/bin/cttsetup.sh
Next, type the following command line in a single continuous line (case-sensitive):
$F90 $F90FLAGS -o lin_eig_self_ex4 lin_eig_self_ex4.f90
$LINK_F90_STATIC -lpthread
The above command line will compile the code lin_eig_self_ex4.f90 into an output binary file
-
7/29/2019 Ultra Users Guide v1.2
27/35
#$ -q short.q
./lin_eig_self_ex4 > output.txt
The following command submits the job submission script:
qsub imsl-submit.sh
When the job is completed without any problems, an output file named output.txt will be created.
Check the output file. If it shows Example 4 of LIN_EIG_SELF is correct, then the job ran
successfully.
-
7/29/2019 Ultra Users Guide v1.2
28/35
3.10. JAVA
The Java SDK is available across all nodes in the cluster.
3.10.1. JAVA Compiler
You can compile your Java codes using the Java compiler installed on the frontend node. You can
test out the example code in the Ultra-samples/java folder in your home directory.
After the code is compiled successfully, you can submit the job using a job submission script:
Example job submission script sendjob1.sh:
#! /bin/bash
#$ -j y
#$ -cwd
#$ -m e
#$ -q short.q
mailto:[email protected]:[email protected]:[email protected] -
7/29/2019 Ultra Users Guide v1.2
29/35
3.11. Quick Summary of Cluster Software
QUICK SUMMARY OF THE SOFTWARE
Software Commands (case sensitive)
MATLAB matlab nodisplay
Gauss tgauss (or) gauss
Gauss Run-time gsrun
R R
STATA SE stata (or) stata-se
ILOG CPLEX cplex
Java Compiler javac
Java Runtime java
Using IMSL (32-bit) $F90 $F90FLAGS -o $LINK_F90_STATIC lpthread
Intel C Compiler(64-bit) /opt/intel/cce/10.1.018/bin/icc -o
Intel C++ Compiler(64-bit) /opt/intel/cce/10.1.018/bin/icpc -o
Intel Fortran Compiler(64-bit) /opt/intel/fce/10.1.018/bin/ifort -o
Intel C Compiler(32-bit) /opt/intel/cc/10.1.018/bin/icc -o
-
7/29/2019 Ultra Users Guide v1.2
30/35
Appendix A.
Linux Commands for Self Administration
A.1. Disk Quota
A.1.1. Check your quota usage using the command quota -s
Example output:
---------------------------------------------------------------------------
[legolas@Ultra legolas]$ quota s
Disk quotas for user legolas (uid 502):
Filesystem blocks quota limit grace files quota limit grace
/dev/hda3 6016M 7813M 9766M 101 0 0
---------------------------------------------------------------------------
Column 2 blocks shows your current quota usage.
Column 3 quota shows the soft limit quota.
-
7/29/2019 Ultra Users Guide v1.2
31/35
[ file or dir name ] : Wildcards (*,?) can be used here to filter
filenames to display.
To create a folder: mkdir [ new foldername ]
To remove a file: rm [ file to delete ]
To remove a folder: rm -rf [ folder to delete ]
To rename a file: mv [ source file name ] [ target file name ]
To change directory: cd [ foldername ]
To move out of folder: cd ..
To move into some folder: cd [ absolute path from root directory ]
To copy a file: cp [ source file ] [ target location ]
To copy a folder: cp -r [ source folder ] [ target location ]
Path ofpresent working directory: pwd
-
7/29/2019 Ultra Users Guide v1.2
32/35
A.3.2. Permissions Handling
Let r/w/x represent numbers 4/2/1 respectively. To turn on the necessary permissions, add the
numbers respectively. Example, r-x is counted as 5.
To set the permission for owner/group/others, use the above number and link them together.
Example, rwxr-x--- is interpreted as 750.
After calculating the numbers from the above 2 steps, type the command to complete the permissions
change:
chmod 755 [ filename ]
This is useful if you want to share a particular folder with a collaborator (contact IITS to create a
common group for both you and the collaborator).
A.4. Miscellaneous Commands
To finding a file, type:
locate [ filename ]
-
7/29/2019 Ultra Users Guide v1.2
33/35
Appendix B.
The vi Text Editor
The vi text editor is one of the text editors available (others include pico, emacs) in the cluster for
editing text files. To use vi from the command line, type either of the following commands:
vi [filename] (opens up the vi editor. Type :help inside to get extensive help)vi +n [filename] (starts editing at line numbern)
VI uses a set of key codes to indicate commands. The following table shows the various commonly
used commands (case-sensit ive):
File Operations Keys
Write to file (save) :wWrite to file and Quit (save and quit) :wq
Write to another filename file :w file
Quit :q
Quit and discard changes :q!
Quit, saving changes if any :x
Text Editing Keys
-
7/29/2019 Ultra Users Guide v1.2
34/35
IITS, Singapore Management University Page 34 of 35
Appendix C.
Cluster Software Availability List
COMPILERS
Software Cmd / Location Documentation Man Pages
GNU C/C++ Compiler
GNU F77 Compiler
gcc / g++
g77
Official Software:
http://www.gnu.org/software/gcc/
Type:
man gcc
man g77
MATHEMATICAL AND STATISTICAL
Software Cmd / Location Documentation Man Pages
GAUSS
- Constrained Max. Likelihood
- Constrained Optimization
- Run-Time Module
tgauss
gauss
gsrun
Official Software:
http://www.aptech.comType:
gauss -h
gsrun -h
MATLAB matlab Official Software:http://www.mathworks.com
Type:
matlab -h
http://www.gnu.org/software/gcc/http://www.gnu.org/software/gcc/http://www.aptech.com/http://www.aptech.com/http://www.mathworks.com/http://www.mathworks.com/http://www.mathworks.com/http://www.aptech.com/http://www.gnu.org/software/gcc/ -
7/29/2019 Ultra Users Guide v1.2
35/35
IITS, Singapore Management University Page 35 of 35
R R Official Software:
http://www.r-project.orgType:
R h
STATA SE stata Official Software Website:
http://www.stata.comType:
stata -h
ILOG CPLEX cplex Official Software Website:
http://www.ilog.comType:
cplex
MATHEMATICAL LIBRARY ROUTINES (FOR USE WITH FORTRAN COMPILERS)
Software Cmd / Location Documentation Man Pages
IMSL Fortran NumericalLibraries (32-bit version) $F90 $F90FLAGS -o example example.f90$LINK_F90_STATIC lpthread Official Software:http://www.vni.comNone
http://www.r-project.org/http://www.r-project.org/http://www.stata.com/http://www.stata.com/http://www.ilog.com/http://www.ilog.com/http://www.vni.com/http://www.vni.com/http://www.vni.com/http://www.ilog.com/http://www.stata.com/http://www.r-project.org/