nccs nccs user forum 24 march 2009. nccs agenda welcome & introduction phil webster, cisto chief...

NCCS

NCCS User Forum

24 March 2009

NCCS

Agenda

Welcome & IntroductionPhil Webster, CISTO Chief

Current System StatusFred Reitz, Operations Manager

NCCS Compute CapabilitiesDan Duffy, Lead Architect

Questions and CommentsPhil Webster, CISTO Chief

User Services UpdatesBill Ward, User Services Lead

NCCS

Key Accomplishments

• Incorporation of SCU4 processors into general queue pool

• Acquisition of analysis system

NCCS

Agenda






NCCS

Key Accomplishments

• SCU4 processors added to the general queue pool on Discover

• SAN implementation

• Improved data sharing between Discover and Data Portal

• RAID 6 implementation

NCCS

Discover Utilization Past Yearby Month

0

20

40

60

80

100M

ar-0

8A

pr-0

8M

ay-0

8Ju

n-08

Jul-0

8A

ug-0

8S

ep-0

8O

ct-0

8N

ov-0

8D

ec-0

8Ja

n-09

Feb-

09

Uti

liza

tio

n P

erc

en

t

Expired

Overhead

SR&T

GISS

GMAO

• 9/4/08 – SCU3 (2064 cores added)• 2/4/09 – SCU4 (544 cores moved from test queue)• 2/19/09 – SCU4 (240 cores moved from test queue)• 2/27/09 – SCU4 (1280 cores moved from test queue)

NCCS

Discover Utilization Past Quarterby Week

0

20

40

60

80

100

30-N

ov-0

807

-Dec

-08

14-D

ec-0

821

-Dec

-08

28-D

ec-0

804

-Jan

-09

11-J

an-0

918

-Jan

-09

25-J

an-0

901

-Feb

-09

08-F

eb-0

915

-Feb

-09

22-F

eb-0

9

Uti

liza

tio

n P

erce

nt

Expired

Overhead

SR&T

GISS

GMAO

• 2/4/09 – SCU4 (544 cores moved from test queue)• 2/19/09 – SCU4 (240 cores moved from test queue)• 2/27/09 – SCU4 (1280 cores moved from test queue)

NCCS

Discover CPU ConsumptionPast 6 Months (CPU Hours)

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

September October November December January February

• 9/4/08 – SCU3 (2064 cores added)• 2/4/09 – SCU4 (544 cores moved from test queue)• 2/19/09 – SCU4 (240 cores moved from test queue)• 2/27/09 – SCU4 (1280 cores moved from test queue)

NCCS

Discover Queue Expansion FactorDecember – February

Eligible Time + Run TimeRun Time

Weighted over all queues for all jobs(Background and Test queues excluded)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

gene

ral_lon

g

debu

g

datam

ove

miss

ion

gmao

dev

gene

ral_hi

gene

ral_sm

all

gene

ral

datam

ove_

l

high_p

riority

ppro

c

visua

l

NCCS

Discover Job Analysis – February 2009Discover Jobs by Job Size - February 2009

0

10,000

20,000

30,000

40,000

50,000

60,000

4 16 32 64 128 256 257+

Discover Total CPU Hours by Job Size - February 2009

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

4 16 32 64 128 256 257+

Discover Expansion Factor by Job Size - February 2009

0

0.5

1

1.5

2

2.5

4 16 32 64 128 256 257+

Discover Jobs by Queue - February 2009

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

45,000

Discover CPU Hours by Queue - February 2009

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1,400,000

1,600,000

1,800,000

2,000,000

Discover Expansion Factor by Queue - February 2009

0

0.5

1

1.5

2

2.5

3

3.5

NCCS

0

2

4

6

8

10

12

Outage Duration

Discover Availability

90.00%

91.00%

92.00%

93.00%

94.00%

95.00%

96.00%

97.00%

98.00%

99.00%

100.00%

December January February

December through February availability• 4 outages

– 2 unscheduled• 0 hardware failures• 1 user error• 1 extended maintenance window

– 2 scheduled

• 11.7 hours total downtime– 1.2 unscheduled– 10.5 scheduled

Outages• 2/11 – Maintenance (Infiniband and GPFS

upgrades, node reprovisioning), 10.5 hours – scheduled outage plus extension

• 11/12 – SPOOL filled due to user error, 45 minutes

• 1/6 – Network line card replacement, 30 minutes – scheduled outage

Mai

ntenan

ce (s

ched

uled p

lus

exte

nsion) –

Infin

iban

d, GPFS u

pgrades

, node

repro

visi

oning

SPOOL fille

d

Network

line

card

mai

ntenan

ce

NCCS

Current Issues on Discover:GPFS Hangs

• Symptom: GPFS hangs resulting from users running nodes out of memory.

• Impact: Users cannot login or use filesystem. System Admins reboot affected nodes.

• Status: Implemented additional monitoring and reporting tools.

NCCS

Current Issues on Discover:Problems with PBS –V

• Symptom: Jobs with large environments not starting.

• Impact: Jobs placed on hold by PBS.

• Status: Awaiting PBS 10.0 upgrade. In the interim, don’t use –V to pass full environment, instead use –v or define necessary variables within job scripts.

NCCS

Future Enhancements

• Discover Cluster– Hardware platform– Additional storage

• Data Portal– Hardware platform

• Analysis environment– Hardware platform

• DMF– Hardware platform– Additional disk cache

NCCS

Agenda






NCCS

FY09 Operating PlanBreakdown of Major Initiatives

• Analysis System Integration– Large scale disk and interactive analysis nodes– Pioneer users in April; full production in June

• FY09 Cluster Upgrade– Two scalable compute units (approximately 4K cores)– Additional 40 TF of Intel Nehalem processors– To be completed by July (subject to vendor availability of equipment)

• Data Portal– Enhance services within the data portal to serve IPCC and other data to the Earth Systems

Grid (ESG) and PCMDI– Actively looking for partners– To be completed by the end of FY09

• Data Management– Concept of operations still being worked out– Actively looking for partners– Plan is to have some amount of capability based on iRODS rolled out by the end of FY09

• DMF Migration from Irix to Linux– Move DMF equipment out of S100 into E100– SGI dropping support for DMF on Irix; will re-use Palm (SGI Linux) system as the new DMF

server– To be completed by June

NCCS

Representative Architecture

ManagementServers

LicenseServers

GPFSManagement

GPFS Disk Subsystems~ 1.3 PB

OtherServices

Analysis FY09Upgrade

~40 TF

FutureUpgrades

TBD

Login

ARCHIVE

DataGateways

Viz

DataPortal

GPFS I/O Nodes

GPFS I/O Nodes

Direct Connect GPFS Nodes

Disk~300 TB

Tape~8 PB

Planned for FY09 Future Plans

Internal Services

Existing

ExistingDiscover

65 TF

GPFS I/O Nodes

PBSServers

DataManagement

NCCS LAN (1 GbE and 10 GbE)

NCCS

Benefits of the Representative Architecture

• Breakout of services– Separate highly available login, data mover, and visualization service nodes– These can be available even when upgrades are occurring within the cluster

elsewhere• Data Mover Service: these service nodes allow for

– Data to be moved between the discover cluster and the archive– Access of data within the GPFS system to be served to the data portal

• WAN accessible nodes within the compute cluster– Users have requested nodes within compute jobs to have access to the

network– The NCCS is currently configuring network accessible nodes to be

scheduled in PBS jobs so users can run sentinel type processes, easily move data via NFS mounts, etc.

• Internal services run on dedicated nodes– Allows for the vertical components of the architecture to go up and down

independently– Critical services are run in a high availability mode– Can even allow for licenses to be served outside the NCCS

NCCS

Analysis Requirements

• Phase 1:– Reproduce Current SGI Capabilities

• Fast access to all GPFS and Archive file systems• FORTRAN, C, IDL, GrADS, Matlab, Quads, LATS4D, Python• Visibility and easy access to post data to the data portal• Interactive display of analysis results

• Beyond Phase 1:– Develop Client/Server Capabilities

• Extend analytic functions to the user’s workstations• Subsetting functions

– In-line and Interactive visualization• Synchronize analysis with model execution• See the intermediate data as they are being generated• Generate images for display back to the user’s workstations• Capture and store images during execution for later analysis

NCCS

DMFArchive

Analysis System Technical Solution

Archive FileSystems GPFS

I/O Servers4 MDS & 16 NSD

Fibre Channel SAN

Multiple Interfaces

Analysis

IB

16 cores 256GB

Discover

Compute

Fibre Channel SAN

IP over IBSingle Stream: 250-300 MB/secAggregate: ~600 GB/sec

NFS, bbftp, scpSingle Stream: 10-50 MB/secAggregate: 1-1.5 GB/sec

Large staging area to minimize data recall from archive

20

Direct GPFS I/O Connections~3 GB/sec per node

Additional Storage

Large Network Pipes

10 GbELAN

NCCS

Analysis System Technical Details

• 8 IBM x3950 Nodes– 4 socket, Quad-core (16 cores per server, 128 cores total)– Intel Dunnington E7440, 2.4 GHz cores with 1,066 MHz FSB– 256 GB memory (16 GB/core)– 10 GbE network interface– Can be configured as a single system image up to 4 servers (64 cores

and 1 TB of RAM)• GPFS File System

– Direct connect I/O servers– ~3 GB/sec per Analysis Node– Analysis nodes will see ALL GPFS file systems, including the nobackup

areas currently in use; no need to “move” data into the analysis system• Additional Disk Capacity

– 2 x DDN S2A9900 SATA disk subsystems– ~900 TB RAW capacity– Total of ~6 GB/sec throughput

21

NCCS

Analysis System Timeline

• 1 April 2009: Pioneer/Early Access Users• If you would like to be one of the first, please let us know.• Contact user services.• Provide us with some details as to what you may need.

• 1 May 2009: Analysis System in Production• Continued support for analysis users migrating off of Dirac.

• 1 June 2009: Dirac Transition• Dirac no longer used for analysis.• Migrate DMF from Irix to Linux.

22

NCCS

Agenda






NCCS

What Happened to My Ticket?

What happened to my ticket?

First, it comes to me at USG for aging…

Then, if it makes it into FootPrints, it will be eaten by trolls…

NCCS

Ticket Closure Percentilesfor the Past Quarter

50%

75%

90%95%

99%

0%

20%

40%

60%

80%

100%

< 0.2 days < 4 days < 31 days < 86 days < 239 days

NCCS

Issue: Commands to Access DMF

• Implementation of dmget and dmput

• Status: resolved– Enabled on Discover login nodes– Performance has been stable since installation

on 11 Dec 09

NCCS

Issue: Parallel Jobs > 1500 CPUs

• Many jobs won’t run at > 1500 CPUs

• Status: resolved– Requires a different version of the DAPL

library– Since this is not the officially supported

version, it is not the default

NCCS

Issue: Enabling Sentinel Jobs

• Need capability to run a “sentinel” subjob to watch a main parallel compute subjob in a single PBS job

• Status: in process– Requires an NFS mount of data portal file

systems on Discover gateway nodes (done!)– Requires some special PBS usage to specify

how subjobs will land on nodes

NCCS

Issue: Poor Interactive Response

• Slow interactive response on Discover

• Status: under investigation– Router line card replaced– Automatic monitoring instituted to promptly

detect future problems– Seems to happen when filesystem usage is

heavy (anecdotal)

NCCS

Issue: Getting Jobs into Execution

• Long wait for queued jobs before launching• Reasons

– SCALI=TRUE is restrictive– Per user & per project limits on number of

eligible jobs (use qstat –is)– Scheduling policy: first-fit on job list ordered by

queue priority and queue time

• Status: under investigation– Individual job priorities available in PBS v10

may help with this

NCCS

Use of Project Shared Space

• Please begin using “$SHARE” instead of “/share” since the shared space may move

• Try to avoid having soft links that explicitly point to “/share/…” for the same reason

NCCS

Dirac Filesystems

• Dirac’s disks are being repurposed for primary archive cache

• Hence, the SGI file systems on Dirac will be going away

• Users will need to migrate data off of the SGI home, nobackup, and share file systems

• Contact User Services if you need assistance.

NCCS

Integrated Performance Monitor (IPM)

• Provides– Short report of resource consumption, and– Longer web-based presentation

• Requires– Low runtime overhead (2%-5%)– Linking with MPI wrapper library (your job)– Newer version of OS for complete statistics

(our job)

NCCS

IPM Sample Output

NCCS

Access to Analysis System

• Pioneer access scheduled for 1 April

• All Dirac analysis users welcome as pioneers

• Initially, no charge against your allocation

• If you have no allocation in e-Books,contact USG and we will resolve

NCCS

Future User Forums

• The next three NCCS User Forums– 23 June, 22 Sep, 8 Dec– All on Tuesday– All 2:00-3:30 PM– All in Building 33, Room H114

• Published– On http://nccs.nasa.gov/– On GSFC-CAL-NCCS-Users

NCCS

Agenda






NCCS

Feedback

• Now – Open discussion to voice your …– Praises– Complaints– Suggestions

• Later to NCCS Support– [email protected]– (301) 286-9120

• Later to USG Lead– [email protected]– (301) 286-2954

nccs nccs user forum 24 march 2009. nccs agenda welcome & introduction phil webster, cisto chief...

Documents