getting started on emerald

Getting Started on Emerald

Getting Started on Emerald

Research Computing Group

its.unc.edu 2

What is Emerald?

Logging into Emerald

File manipulation on Emerald

Submitting jobs on Emerald

Interactive programs on Emerald

Course OutlineCourse Outline

its.unc.edu 3

Help DocumentationHelp Documentation

Getting Started on Emerald•http://help.unc.edu/6020

•General overview of Emerald for range of users

Short Course – Getting Started on Emerald•http://help.unc.edu/6479

•Detailed notes for beginning Emerald users

its.unc.edu 4

What is Emerald?What is Emerald?

352-processor Linux Cluster•Maintained by Research Computing Group

Appropriate for all users regardless of expertise level

Other Servers:•Cedar/Cypress (128-processor SGI/Altix)

•Topsail (4160-processor Dell Linux Cluster)

Mass Storage•Account access

its.unc.edu 5

Advantages of Using Emerald

Advantages of Using Emerald

High performance

Large capacity

Parallel processing

Many available software packages

Variety of compiling options

its.unc.edu 6

Emerald Linux ClusterEmerald Linux Cluster

its.unc.edu 7

Distributed vs. Shared MemoryDistributed vs.

Shared Memory

Shared memory - single address space. All processors have access to a pool of shared memory. (examples: Yatta, Cedar/Cypress) Methods of memory access : Bus and Crossbar

Distributed memory - each processorhas it’s own local memory. Must do message passing to exchange data between processors. (examples: Emerald, Topsail)

MEMORY

BUS

CPU CPU CPU CPUCPU CPU CPU CPU

MMMM

NETWORK

its.unc.edu 8

Logging Into EmeraldLogging Into Emerald

UNIX/Linux/OSX

•ssh my_onyen@emerald.unc.edu

Windows: SSH Secure Shell

•Setting up a Profile for Emerald

•Forwarding X11 packets

its.unc.edu 9

Home and Work Directories on

Emerald

Home and Work Directories on

Emerald Home Directory

• /afs/isis/home/m/y/my_onyen/•250MB quota•~/private/•Files backed up daily [ ~/OldFiles ]•Space quota/usage in Home Directory:

fs lq

Work Directory• /netscr/my_onyen/•No space limit but periodically cleaned

its.unc.edu 10

File Manipulation Commands

File Manipulation Commands

SSH Secure File Transfer Copy files

• cp command

cp /afs/isis/depts/atn/rcg/example_code/Gaussian/water.com /netscr/my_onyen/.

cp /afs/isis/depts/atn/rcg/example_code/sas/test.sas /netscr/my_onyen/.

• cp -r cp -r ~/private/TestDirectory .

Move files• mv commands

mv ~/private/testfile.txt .

its.unc.edu 11

File Manipulation Commands

File Manipulation Commands

Tar archive

•To create a tar filetar –cvzf TestDirectory.tgz ./

•To see a tar file’s table of contenttar –tvzf TestDirectory.tgz

•To untar a tar file tar –xvzf TestDirectory.tgz

its.unc.edu 12

Submitting Jobs: LSF and Packages

Submitting Jobs: LSF and Packages

LSF (Load Sharing Facility)•Fairly distribute compute nodes among

users•60 processor per user limit

Packages• ipm commands

ipm add (ipm a) ipm remove (ipm r) ipm query (ipm q)

•Available packageshttp://www.renci.org/unc/computing/resources/

applications.php

its.unc.edu 13

Details of LSFDetails of LSF

Submission host

LIM

Batch API

Master host

MLIM

MBD

Execution host

SBD

Child SBD

LIM

RES

User jobLIM – Load Information ManagerMLIM – Master LIMMBD – Master Batch DaemonSBD – Slave Batch DaemonRES – Remote Execution Server

queue1

2

3

45

6 7

89

10

11

12

13

Loadinformation

otherhosts

otherhosts

bsub app

its.unc.edu 14

Submitting Jobs: bsub Command

Submitting Jobs: bsub Command

bsub command All files must be in /netscr/my_onyen/ bsub [- bsub_opts] executable [-exec_opts]

Queues – bqueues command

•week

•idle

bsub –o

•bsub –o out.%J

its.unc.edu 15

Submitting Jobs: Following Job

Progress

Submitting Jobs: Following Job

Progress bjobs

•bjobs –l JobID

•Shows current status of job

bhist•bhist –l JobID

bkill•bkill JobID

•Ends job prematurely

bfree

its.unc.edu 16

Submitting Jobs: Specialty ScriptsSubmitting Jobs: Specialty Scripts

bsas•bsub -q week -R blade sas program.sas

•bsas test.sas

bmatlab•bsub -q week -R blade matlab -nodisplay

-nojvm -nosplash program.m -logfile program.log

•bmatlab test.m

its.unc.edu 17

Compiling on EmeraldCompiling on Emerald

• Compilers

– FORTRAN 77/90/95

– C/C++

• Parallel Computing – OpenMP

– MPI (MPICH, LAM/MPI, MPICH-GM)

its.unc.edu 18

Compiling Details on Emerald

Compiling Details on Emerald

Compiler Package name Command

Intel intel_fortran, intel_CC ifort, icc

Portland Group pgi pgf77, pgf90,pgcc,pgCC

Absoft profortran f77, f90

GNU gcc g77,gcc, g++

its.unc.edu 19

Compiling Details on Emerald

Compiling Details on Emerald

Add a compiler into your working environment

•ipm add package_name

Compile a code

•command code.f –o executable

Run executable on a compute node using the bsub command

•bsub –q week –R blade executable

its.unc.edu 20

Submitting Jobs: Job Output

Submitting Jobs: Job Output

Output sent to email

•my_onyen@email.unc.edu

bsub –o

•Output saved in working directory

bsub –u email@address

•Output sent to specified email address

its.unc.edu 21

Interactive Jobs: Setup

Interactive Jobs: Setup

X-Windows

•Linux/OSXX11 client

•WindowsX-Win32

Offered on UNC Software Acquisition sitePort forwarding on SSH Secure Shell

Setting up a session on X-Win32

its.unc.edu 22

Interactive Jobs: Submission

Interactive Jobs: Submission

–Ip

•bsub –q int –R blade –Ip sas

•bsub –q int –R blade –Ip gv

Specialty Scripts

•xsas

•xstata

its.unc.edu 23

Contacting Research Computing

Contacting Research Computing

For assistance with Emerald, please contact the Research Computing Group:

•Email: research@unc.edu

•Phone: 919-962-HELP

•Submit help ticket at http://help.unc.edu