overview of uge · ① log into the gateway node (gw.ddbj.nig.ac.jp) ②execute qlogin and log into...

Explanatory meeting for users of supercomputer system -- Overview of UGE --

June 26, 2012

What is Univa Grid Engine (UGE)?

It is software that is used to construct a grid computing system. It functions as a batch job system.

It is a commercial product derived from Sun Grid Engine 6.2U5 (the last open-source version). The main developers of SGE participated in the development of UGE.

The commands, etc., for entering jobs in the UGE are the same as that in SGE.

1

Advantages of using UGE Multiple jobs can be smoothly executed in a

sequential manner.

When more than one user enter several jobs simultaneously, UGE carries out the scheduling.

Effective scheduling is carried out according to the memory, CPU, etc., required by a job.

2

Precautions for using UGE Parallelization of jobs and other such

functions cannot be performed.

If the resource demand when a job is entered is not declared properly, large-scale hang-up of computer may occur.

How to use a supercomputer

3

① Log into the gateway node (gw.ddbj.nig.ac.jp) ②Execute qlogin and log into the interactive node ③ Enter a job from the host for which qlogin is executed into UGE ④ In the UGE, the job is executed at a node with small load ⑤ The job execution result is output to the home directory, lustre ⑥ The job execution result is checked

Job Job Job

Result Result Result Result Result

①

③

④

gateway

②

⑥ ⑤

1 unit (active-standby) dedicated to communications with external

8 units for professional use, 8 units for research dedicated to interactive

81 units for professional use, 208 units for research dedicated to batch

Basic terms (concept) (1)

Host (node) Physically existing computer

Master host Host in which master daemon (qmaster) of UGE operates The master daemon is the daemon that controls the UGE, which carries out acceptance of job, scheduling, delivery to execution host, collection, etc.

Execution host Host in which the execution daemon (execd) of UGE operates The execution daemon accepts the instruction of job execution from the master daemon and executes the job.

4

qmaster

execd

execd

execd

execd

Job Job Job

Basic terms (concept) (2)

Submit host Host that can enter a job into UGE The execution host for which login is enabled using the qlogin command is a submit host

Queue Target of job entry

It is configured using more than one execution host. There are several types of queues depending on their application.

Job slot “Container” for executing a job that is set for each execution host.

A job is entered into a queue and finally stored in the slot.

5

submit host

execd

execd

execd

execd

execd

execd

execd

JobSlot JobSlot

JobSlot JobSlot

JobSlot JobSlot

JobSlot JobSlot

qmaster

×

Job

Job

Two UGE environments

The supercomputer system has two UGE environments as shown below.

The usable environmental setting is configured at login, so a user does not need to configure the settings.

6

UGE environment for DDBJ operation UGE environment that can be used with account for DDBJ operation SGE_ROOT=/home/geadmin/UGES SGE_CELL=uges

UGE environment for research UGE environment that can be used with account for general research SGE_ROOT=/home/geadmin/UGER SGE_CELL=uger

Types of queues (for research, as of June 26)

Queue name Number of job slots

Upper limit of execution time (days)

Application

week_hdd.q 1600 14 If queue/resource is not specified, a job is entered in this queue

week_ssd.q 832 14 Used when a job that may take short execution time is executed using ssd

month_hdd.q 96 62 Used when a job that may require long execution time is executed

month_ssd.q 64 62 Used when a job that may take long execution time is executed using ssd

month_gpu.q 992 62 Used when a job that uses gpu is executed

month_medium.q 160 62 Used when a job that uses a medium node is executed

month_fat.q 768 62 Used when a job that uses a fat node is executed

debug.q 64 1 Used for checking the operation of a job

login.q 128 － Used for entering a job

7

Types of queues (for commercial use, as of June 10)

Queue name Number of job slots

Upper limit of execution time (days)

Application

month_hdd.q 448 62 If queue/resource is not specified, a job is entered in this queue

debug.q 32 1 Used for checking the operation of a job

login.q 64 － Used for entering a job

8

Upper limit of execution time

The upper limit of execution time is set for resolving jamming of jobs that are waiting for execution when a jam is generated. A job that exceeds the upper limit of the execution time is killed.

The execution time is computed as the actual time after a job is executed. (It is not the CPU use time. The time of waiting in a queue is not included.)

Before entering a job, it is necessary to know the execution time using the environment for checking the operation.

9

qlogin

10

$ qlogin

Your job 329 ("QLOGIN") has been submitted

waiting for interactive job to be scheduled ...

Your interactive job 329 has been successfully scheduled.

Establishing builtin session to host t217i ...

$ uname -n

t217

When a job is entered, log into the host that has sufficient resource of login.q using the qlogin command from the gateway host (common to environment for both research and DDBJ operation).

When logging into the execution host, ensure that you use the qlogin command to do so. Do not directly log into the system to execute a job, as the load dispersion mechanism would not function properly. (*A user who directly logs into the system is recorded.)

Job entry (1)

11

#!/bin/sh

#$ -S /bin/sh

pwd

hostname

date

sleep 20

date

echo "to stderr" 1>&2

“#$” at the second line is the prefix for specifying the UGE option. The interpreter that is used when this shell script is operated on UGE is specified with “#$ -S” (in this example, the interpreter is /bin/sh). If this line is omitted, it is necessary to specify “-S path of interpreter to be used” with the command option when a job is entered.

A job is entered by creating a shell script that is described for UGE. An example is shown below (file name: test.sh).

Job entry (2)

12

$ qsub test.sh

Enter a job using the qsub command

When a job is entered, it is inserted in the queue that waits for execution.

The status of the entered job can be checked using the qstat command (to be described).

After execution, check the output of the job. A file in which the standard output of job and the standard error output are recorded is output to the home directory.

$ cat ~/test.sh.o325

/lustre1/home/ddbjuser

t165

2012年 3月 21日水曜日 11:15:01 JST

2012年 3月 21日水曜日 11:15:21 JST

$ cat ~/test.sh.e325

to stderr

Main options of qsub (1)

13

-S <path of interpreter>

Specify the path of the interpreter when the script file is executed. The interpreter of the script language, such as Perl or Ruby, can be specified in addition to the shell, e.g., (sh is specified): -S /bin/sh (Perl is specified): -S /usr/local/bin/perl

-cwd

A job is executed not in the home directory but in the directory where the qsub command is executed. If this option is specified, the file containing the standard output and standard error output are output to the directory where qsub command is executed.

-o <output destination of standard output> -e <output destination of standard error output>

Specify the output destinations of the standard output of job and the standard error output. If the standard output or standard error output is not output as a file, specify “/dev/null” as the output destination, e.g., -o /dev/null -e /dev/null

Main options of qsub (2)

14

-N <alias name of job>

The name of job that can be checked using qstat is changed to the specified name. If it is not specified, the job name is the same as the script name.

-l resource demand 1,resource demand 2,…

They are mainly used for selecting a queue or changing the upper limit of memory usage capacity. Details are described later.

-l resource demand 1 –l resource demand 2 –l …

Checking job status

15

The status of an entered job is checked using the qstat command

$ qstat

job-ID prior name user state submit/start at

-------------------------------------------------------------

325 0.00000 test.sh ddbjuser qw 03/19/2012 19:11:56

If a job exists in the waiting queue, “qw” is displayed as the state

$ qstat

job-ID prior name user state submit/start at

-------------------------------------------------------------

325 0.00000 test.sh ddbjuser r 03/19/2012 19:11:56

While a job is executed, "r" is displayed for state

r A job is being executed on the execution host

qw A job is waiting in a queue

t A job is being transferred to the execution host

E An error is generated in a job

d A job is being deleted

The main states are shown below. States can also be displayed in combinations of more than one.

Main options of qstat

16

-f

The queue usage status is displayed additionally. e.g., qstat –f

-u [uid]

The specified job of [uid] is also displayed. If it is set to "‘*’", the jobs of all users are displayed. e.g., qstat –u ‘*’

-j [jobid]

The detailed information of specified job of [jobid] is checked. The reason of error status “Eqw” can be checked. e.g., qstat –j 325

Deletion of a job

17

$ qsub test.sh

Your job 326 ("test.sh") has been submitted

$ qdel 326

ddbjuser has deleted job 326

The qdel command is used to delete a job. A job is deleted by specifying the job ID or UID.

$ qsub test.sh


$ qsub test.sh


$ qdel -u ddbjuser

ddbjuser has registered the job 327 for deletion

ddbjuser has registered the job 328 for deletion

When the job ID is specified (specify only job ID)

When UID is specified (UID is specified using “-u” option)

Checking job execution result

18

The details of a job that has been executed are checked using the qacct command. The resource that was actually consumed by the job, etc., can be checked.

$ qacct -j 325

==============================================================

qname week_hdd.q

hostname t165i

group se

owner ddbjuser

project NONE

(*An omission*)

cpu 0.032

mem 0.001

io 0.000

iow 0.000

maxvmem 208.207M

arid undefined

Precautions before entering a job

19

* Before entering multiple jobs, ensure that a test is conducted. >> If the memory is insufficient, several hosts can cause a hang-up >> A lot of error jobs can cause overload of UGE

•Do not allocate or output an input file or a file of final output to a directory of each local host such, as /tmp and /ssd >> The input cannot be read by a host in which a job is being executed >> The result cannot be referenced after execution

*1 The number of jobs that are executed by a job at the same time should be 1. (Do not fork multiple processes with one job without using def_slot (to be described)) *1 The number of threads of a process should be 1. (Do not execute a process with multiple threads without using def_slot (to be described)) >> Load dispersion cannot occur and the host causes a hang-up

Switching usage of queue (for research) (1)

20

$ qsub test.sh

Do not specify a resource

week_hdd.q,week_ssd.q is used The priority order is week_hdd.q > week_ssd.q

$ qsub –l month test.sh

Specify "month" (* Specify it when a long calculation time is estimated)

month_hdd.q,month_ssd.q,month_gpu.q is used The priority order is month_hdd.q > month_ssd.q > month_gpu.q

$ qsub –l ssd test.sh

Specify "ssd" (* Specify it when a job that uses SSD is entered)

Only week_ssd.q is used

The usage of a queue can be switched by specifying a resource with the "-l" option


21

$ qsub –l month –l ssd test.sh

Speicfy "month" and " ssd" (* Specify them when a long calculation time is estimated for a job that uses SSD)

month_ssd.q,month_gpu.q is used The priority order is month_ssd.q > month_gpu.q

$ qsub –l month –l gpu test.sh

Specify "month" and " gpu" (* Specify them when a job that uses GPU is entered)

Only month_gpu.q is used * When a host on which GPU is mounted is used, ensure that "-l month“ is specified * Only one job that requires GPU can operate on one host on which GPU is mounted

$ qsub –l month –l medium test.sh

Specify "month" and " medium" (* Specify them when a job that uses a medium node is entered)

Only month_medium.q is used * When a medium node is used, ensure that "-l month“ is specified


22

$ qsub –l month –l fat test.sh

Specify "month" and " fat" (* Specify them when a job that uses a fat node is entered)

Only month_fat.q is used * When a fat node is used, ensure that "-l month“ is specified

$ qsub –l debug test.sh

Specify "debug" (* Specify it when checking the operation of a job)

debug.q is used

$ qsub –l debug –l gpu test.sh

Specify "debug" and "gpu" (* Specify them when checking the operation of a job that uses GPU)

A host on which GPU is mounted in debug.q is used


23

*Caution* To use a GPU, medium, or fat node, ensure that you specify "month" in addition to the resources ("gpu,“ "medium," "fat,“ respectively). With the current queue configuration, all of the GPU, medium, and fat nodes are allocated only for a queue for a time-consuming calculation, so it is necessary to specify the resource "month" for using those queues. If "month" is not specified, the current queues have no resource that satisfies the resource specification condition, and so the submission is normally carried out, but the job is not executed.

$ qsub –l debug test.sh

Specify "debug" (* Specify it when checking the operation of a job)

debug.q is used

Switching usage of queue (for professional use)

24

$ qsub test.sh

Do not specify a resource

month_hdd.q is used

When large amount of memory is used (1)

The memory available for a UGE job is restricted to 4 GB by default.

When a large amount of memory is used, declare the memory usage using the "-l" option

25

$ qsub –l s_vmem=8G –l mem_req=8G test.sh

When 8 GB of memory is used for 1 job

$ qsub –l s_vmem=128G –l mem_req=128G –l month –l medium test.sh

When 128 GB of memory is used for 1 job on a medium node

When large amount of memory is used (2)

26

s_vmem: Declares the upper limit value of virtual memory that can be used by a job. The job cannot use the memory exceeding the amount specified here.

mem_req: Declares the memory to be used. For the execution host, the value of "mem_req" is set as an index that indicates the remaining amount of memory, and the value increases or decreases depending on the job execution status. It is used as an index of load dispersion.

qmaster

Job

mem_req=8G

Job

mem_req=128G

execd

mem_req=64G

execd

mem_req=2048G

execd

mem_req=64G

execd

mem_req=1920G

execd

mem_req=56G

Job

mem_req=8G

Job

mem_req=128G

1. While a job is being executed, mem_req value of the host decreases for the portion declared by the job

2. If the job is terminated, mem_req for the portion declared by the job is reset to the original value

3. If mem_req of the host is smaller than mem_req declared by a job, the job is not executed for the host

Array job (1)

27

$ cat arraytest.sh

#!/bin/sh

#$ -S /bin/sh

echo ---

echo JOB_ID: ${JOB_ID}

echo SGE_TASK_ID: ${SGE_TASK_ID}

echo SGE_TASK_FIRST: ${SGE_TASK_FIRST}

echo SGE_TASK_LAST: ${SGE_TASK_LAST}

echo SGE_TASK_STEPSIZE: ${SGE_TASK_STEPSIZE}

echo ---

$ qsub –t 1-6:2 arraytest.sh

Your job-array 1031.1-6:2 ("arraytest.sh") has been submitted

$ qstat

job-ID prior name user state submit/start at queue slots ja-

task-ID

-----------------------------------------------------------------------------------------------------------

------

1031 0.50000 arraytest. ddbjuser r 03/19/2012 00:43:13 week_hdd.q@t168i 1 1



If a job is entered as an array job, it is possible to give a different parameter for the same job and execute it repeatedly. If the "-t" option of qsub is used, the array job can be entered.

Array job (2)

28

$ ls arraytest.sh.o1031.*

arraytest.sh.o1031.1 arraytest.sh.o1031.3 arraytest.sh.o1031.5

$ cat arraytest.sh.o1031.1

---

JOB_ID: 1031

SGE_TASK_ID: 1

SGE_TASK_FIRST: 1

SGE_TASK_LAST: 6

SGE_TASK_STEPSIZE: 2

---

$ cat arraytest.sh.o1031.5

---

JOB_ID: 1031

SGE_TASK_ID: 5

SGE_TASK_FIRST: 1

SGE_TASK_LAST: 10

SGE_TASK_STEPSIZE: 2

---

To prevent UGE overload, the number of jobs that can be entered by one user is limited. If you try to enter jobs exceeding this limit, an error is generated and you cannot enter the jobs. The upper limit of jobs to be entered per user for this system is 5000. If a job is entered as an array job, the load on UGE can be reduced. If array jobs of 5000 jobs are entered, the jobs for 5000 * SGE_TASK_ID can be executed. The upper limit of SGE_TASK_ID is 75000.

MPI job (1)

29

$ cat mpitest.sh

#!/bin/sh

#$ -S /bin/sh

#$ -pe mpi 2-24

#$ -cwd

/usr/local/bin/mpirun -np $NSLOTS ¥

-machinefile $TMPDIR/machines ¥

./mpitest

An example of shell script that is used to enter an MPI job is shown below.

-pe <MPI execution environment name> <minimum parallel count>-<maximum parallel count>

Specify MPI execution environment (to be described), minimum parallel count, and maximum parallel count

$NSLOTS

Values that are automatically decided from [minimum parallel count] to [maximum parallel count] shown above are set according to the free space of queue

-machinefile $TMPDIR/machines

The file $TMPDIR/machines is automatically generated by UGE

MPI job (2)

30

$ qsub mpitest.sh

Your job 1292 ("mpitest.sh") has been submitted

$ qstat

job-ID prior name user state submit/start at queue

slots ja-task-ID

-----------------------------------------------------------------------------------

------------------------------

1292 0.50000 mpitest.sh ddbjuser r 03/19/2012 20:55:24 week_hdd.q@t303i

24

$ cat mpitest.sh.o1292

Hellow World from Process 0 of 24 running on t303


(*中略*)


Enter an MPI job in UGE

Main MPI execution environment mpi: executes parallel jobs using as many hosts as possible mpi-fillup: executes parallel jobs using the same host whenever possible

Use of parallel environment def_slot (1)

31

Use it when you enter a job that may cause overload if it is entered directly, such as a job that forks multiple processes or a job that executes a multi-thread process.

$ qsub –pe def_slot 2 test.sh

The number of job slots consumed by this job is redefined by a value following "def_slot“. In this example, this job consumes two job slots. It is used as an index of a value that specifies the maximum number of processes that are simultaneously initialized in the target job, and the maximum number of threads used by a process initialized in the job.

Use of parallel environment def_slot (2)

32

*Cautions* If def_slot is specified, the amount of resource demand becomes "resource amount specified by -l" × "number of slots specified by def_slot" Note that the excessive resource may be demanded unintentionally. If one of the following options is specified, the amount of resource demand becomes 32 GB.

$ qsub –pe def_slot 4 –l max_vmem=8G –l mem_req=8G test.sh

$ qsub –pe def_slot 4 test.sh

If the resource demand is not clarified, the default value is applied. In the following case, the amount of resource demand becomes 16 GB.

$ qsub –pe def_slot 10 –l max_vmem=8G –l mem_req=8G test.sh

In the following case, the amount of resource demand becomes 80 GB. However, a thin node does not have a node that meets the condition, so the job is not executed if it is submitted.

Contact information

If you have any enquiries or opinions, please contact

Supercomputer SE team, National Institute of Genetics

Mail: [email protected]

Room: w202

Extension: 9461

http://www.ddbj.nig.ac.jp/system/supercom/supercom-intro.html

33

mailto:[email protected]



34

Revision history

Date of revision Revised content

March 21, 2012

Newly created

May 10, 2012 The number of job slots for "type of queue" is corrected according to the current situation. Precautions on use of def_slot are added.

June 18, 2012 Upper limit of execution time for month_*.q is changed to 62 days from 31 days, “type of queue” is corrected.

June 26, 2012 Configuration of the queues is changed, “type of queue” is corrected.

The types of queues for professional use are reduced, “Switching usage of queue (for professional use)” is modified.

overview of uge · ① log into the gateway node (gw.ddbj.nig.ac.jp) ②execute qlogin and log into...

Documents