FACULTY OF CUMPUTER SCIENCE & ENGINEERING

Ghulam Ishaq Khan Institute of Engineering Sciences & Technology

User Manual For

HPC Cluster at GIKI

Volume

1.0

Designed and prepared by

Faculty of Computer Science &

Engineering (FCSE), GIKI

Ghulam Ishaq Khan Institute of Engineering Sciences & Technology 23640, Topi, Swabi

Khyber Pukhtunkhwa Phone (0938) 271858 • Fax (0938) 271877

Table of Contents

Section 1: Introduction ........................................................ 3

Section 2: Accessing The Cluster @ GIKI ......................... 4

2.1 Linux Client .......................................................................... 4

2.2 Windows Client .................................................................... 5

2.2.1 Password-less Authentication ................................................. 6

Section 3: Submitting Jobs on the Cluster ........................ 8

3.1 Running Sequential Jobs .................................................... 8

3.2 Running Parallel MPI Jobs .................................................. 9

3.3 Running Parallel MPJ Express Jobs ................................ 10

Contact Information ........................................................... 12

Contact Address ................................................................ 12

3

Section 1: Introduction

The High Performance Computer or HPC cluster at GIKI uses supercomputer and parallel processing for solving complex computational problems rapidly and efficiently. The HPC cluster at GIKI has the following specifications.

1. Total CPUs: 160 cores using latest 16C AMD Opteron Processors

2. Total main memory: 640 GB of 1600MHz

3. Total GPUs: 1024 cores (Tesla 2090 system)

4. Connectivity: 10GB Ethernet switch

5. Hard disks: 1.8TB with 15k rpm (speed)

Open source compilers such as mpicc, mpic++, mpif90, mpif77 are installed along with gcc (4.4.6) in rocks cluster 6.1. A block diagram of the HPC cluster at GIKI is shown below

The above information and the block diagram of HPC cluster at GIKI were provided by Dr. Masroor Hussain (Assistant Professor at Faculty of Computer Science and Engineering, GIKI)

The IDs of the nodes are:

Master node: hpc.giki.edu.pk

Slave 1: Compute_0_0

Slave 2: Compute_0_1

Slave 3: Compute_ 0_3

Section

The cluster can be accessed either through a secure connection on Linux or through a Windows client

2.1 Linux Client

Type the following on shell prompt

$ ssh u30@hpc.giki.edu.pk

or you can also type

$ ssh u30@

The following messages appear

You will be prompted for your password. Enter your password in order to gain access into the cluster. The password will not be displayed for security reasons

After a successful login, the following messages will appear. allotted to you. The username can be requested by sending an application to hpc@giki.edu.pk

4

Section 2: Accessing The Cluster @

The cluster can be accessed either through a secure connection on Linux or through a Windows client.

Linux Client

Type the following on shell prompt and press enter

@hpc.giki.edu.pk

can also type

$ ssh u30@192.168.100.21

following messages appear

You will be prompted for your password. Enter your password in order to gain access into the cluster. The password will not be displayed for security reasons

After a successful login, the following messages will appear. You will see the username allotted to you. The username can be requested by sending an application to hpc@giki.edu.pk

Accessing The Cluster @ GIKI

You will be prompted for your password. Enter your password in order to gain access into the cluster. The password will not be displayed for security reasons

You will see the username allotted to you. The username can be requested by sending an application to

5

2.2 Windows Client

1. In order to gain access to the cluster through a windows client you can use an ssh client such as PuTTY. PuTTY is an open source program that can be downloaded from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

2. After you have downloaded PuTTY, then in order to run it, double-click on it and type in the following IP address in the hostname / IP address box

192.168.100.21 or hpc.giki.edu.pk

3. Select SSH as the connection type and 22 as the port number

4. Click open to start your session

5. You will be prompted for your username. Type in your username and press enter.

For acquiring a username and password, contact hpc@giki.edu.pk

6. Now you will be prompted for the password. Type in your password and press enter. The password will not be displayed due to security reasons and the following messages will appear

6

2.2.1 Password-less Authentication

The first step is to generate an SSH key pair. This private / public key pair will be used to authenticate you on the target machine. To generate this key, type the following in a terminal:

$ ssh-keygen –t rsa

After doing so the command will ask you several questions:

Enter file in which to save the key (/export/home/msuleman/.ssh/id_rsa):

(you can press enter here) Enter passphrase (empty for no passphrase):

(you can press enter here) Enter same passphrase again:

(you can press enter here)

7

The next step will be to copy the public key to a file names authorized_keys. To do this, type the following commands:

$ cd ~ $ cat ./.ssh/id_rsa.pub >> ./.ssh/authorized_keys

8

Section 3: Submitting Jobs on the Cluster

3.1 Running Sequential Jobs

1. In order to launch the application, create the following shell script

#!/bin/bash #seq.sh # # --------------------------- # The following will be the name of the Sequential Job. You may change it according to your needs. #$ -N SEQUENTIAL_Job

# The output and errors of the programmes will be written in # SEQUENTIAL_Job.oJOB_ID and SEQUENTIAL_Job.eJOB_ID respectively. #Path to the executable. /usr/bin/top # End of Script

2. Set execute permissions to the script

$ chmod 755 seq.sh

3. Now submit your command by performing following command.

$ qsub -V seq.sh

You will see your job id once this command is executed

4. Enter the following command to view the status information about your job.

$ qstat

5. To delete your running job issue the command:

$ qdel 30

Where 30 is your job id

9

3.2 Running Parallel MPI Jobs

1. Create the following shell script for launching an application

#!/bin/bash #mpi.sh # --------------------------- # The following will be the name of the MPI Job. You may change it

#according to your needs. #$ -N MPI_Job # The output and errors of the programmes will be written in # MPI_Job.oJOB_ID and MPI_Job.eJOB_ID respectively. # Number of slots/cores that need to be utilized are given in the #following line. Currently it shows 2-8 i.e. The number of free

cores #will be assigned to the programme. #$ -pe mpi 2-8 #The parallel environment that is being used is mpi # --------------------------- # needs in # $NSLOTS # the number of tasks to be used/cores assigned by grid engine

# $TMPDIR/machines # a valid machine file to be passed to mpirun echo "Got $NSLOTS slots." mpirun -np $NSLOTS -mca ras gridengine --hostfile $TMPDIR/machines ./a.out # End of Script

2. Set execute permissions to the script by typing following command

$ chmod 755 mpi.sh

3. To submit your command, type the following command.

$ qsub -V mpi.sh

This also prints out the job id of your submitted job

4. To view the status information of your job, enter the following command

$ qstat

5. In order to delete a running job type in the following command.

$ qdel 30

Where 30 is your job id.

10

3.3 Running Parallel MPJ Express Jobs

1. Create a shell script for launching an application

#!/bin/bash #mpjrun_sge.sh # # --------------------------- # The following will be the name of the MPJ Express Job. You may change it according to your needs. #$ -N MPJ_Job # The output and errors of the programmes will be written in # MPJ_Job.oJOB_ID and MPJ_Job.eJOB_ID respectively. # Number of slots/cores that need to be utilized are given in the following line. Currently it shows 2-8 i.e. The number of free cores will be assigned to the programme. #$ -pe mpj 2-8 #The parallel environment that is being used is mpj # --------------------------- # # needs in # $NSLOTS # the number of tasks to be used/cores assigned by grid engine # $TMPDIR/machines # a valid machine file to be passed to mpirun echo "Got $NSLOTS slots." cat $TMPDIR/machines > `pwd`/machines # if your d_port, s_port and mpj_port are different than default then you have to give them properly in the respective command mpjboot $TMPDIR/machines mpjrun.sh -np $NSLOTS -dev niodev World mpjhalt machines rm `pwd`/machines # End of Script

2. Set execute permissions to the script

$ chmod 755 mpjrun_sge.sh

3. Now submit your command by performing following command.

$ qsub -V mpjrun_sge.sh

This will also print out the job id of your submitted job.

4. Enter the following command to view your job’s status.

$ qstat

11

5. To delete your running job, type in the following command

$ qdel 30

Where 30 is your job id.

Running MPI Jobs using hpc.giki.edu.pk

1. Creating a shell script for launching the application

#!/bin/bash #mpi.sh # # --------------------------- # The following will be the name of the MPI Job. You may change it #according to your needs. #$ -N MPI_Job # The output and errors of the programmes will be written in # MPI_Job.oJOB_ID and MPI_Job.eJOB_ID respectively # Number of slots/cores that need to utilized are given in the # following line. Currently it shows 2-8 i.e. The number of free cores # will be assigned to the programme. #$ -pe mpi 2-8 # The parallel environment that is being used is mpi # --------------------------- # # needs in # $NSLOTS # the number of tasks to be used/cores assigned by grid engine # $TMPDIR/machines # a valid machine file to be passed to mpirun echo "Got $NSLOTS slots." mpirun -np $NSLOTS -mca ras gridengine –mca btl mx,sm,self -- hostfile $TMPDIR/machines ./a.out # You can also you –mca mtl mx,sm,self instead of –mca btl mx,sm,self # End of Script

2. Set execute permissions to the script

$ chmod 755 mpi.sh

3. Now submit your command by performing following command.

$ qsub -V mpi.sh

This will also print out the job id of your submitted job

4. Enter the following command to retrieve status information about your job.

12

$ qstat

5. To delete your running job issue the command:

$ qdel 30

Where 30 is your job id.

Contact Information

In case of an inquiry, send a query at:

hpc@giki.edu.pk

hussain@giki.edu.pk

msuleman@giki.edu.pk

Contact Address

Faculty of Computer Science and Engineering, Ghulam Ishaq Khan Institute of Engineering Sciences & Technology 23640, Topi, Swabi Khyber Pukhtunkhwa Phone (0938) 271858 Fax (0938) 271877