Scientific Computing Division

NCAR/CISL/SCDUser Forum

Using the Mass Store Subsystem

Efficiently

May 19, 2005

Mike Page

Consulting Services Group

mpage@ucar.edu

Scientific Computing Division

Efficient Usage

NCAR\ Users CISL\ SCD

Scientific Computing Division

NCAR Mass Store Subsystem(MSS)

• Currently stores two petabytes of data• Library of Congress (printed collection)

• 10 Terabytes

• Mass Store holds 200 * Library of Congress

• Data holdings increasing exponentially• 1986 - 2 Tb• 1997 - 100 Tb• 2002 - 1000 Tb• 2004 - 2000 Tb

Mass Store Holdings

0

500

1000

1500

2000

2500

1985 1990 1995 2000 2005

Terabytes

Scientific Computing Division

NCAR Mass Store Subsystem Capacity vs. Demand

• Planning, planning, planning • CSG/MSSG assistance available

• Is it a file server?• Recommended file sizes• Recommended volume of metadata operations

• Other keys to efficient usage• Be selective in the data stored on the MSS• Avoid repetitive reads/writes• Choose class of service and retention periods

according to value of data

Scientific Computing Division

Scientific Computing Division

If: – you are no longer active on any project– your project closes– you are no longer employed by UCAR/NCAR– you are doing periodic maintenance of your MSS files

You can:– delete your MSS files yourself– request that the SCD delete your files– change the project number to an active account– transfer ownership of the files to another user – transfer the data between the MSS and other media

– 1 Terabyte > 300 DVDs– transfer data to another network location

File purge policies

Scientific Computing Division

How to access information on your MSS holdingshttp://www.scd.ucar.edu/docs/mss/MSSholdings.html

Scientific Computing Division

MSS Summary Report

Scientific Computing Division

Summaries

Scientific Computing Division

Class of Service

Scientific Computing Division

Mass Store Access from Command Line and Batch Scripts• msreadmsread

• msread [-eof] [-f fmt] [-l n] [-nomail] [-noretry] [-nowait] \msread [-eof] [-f fmt] [-l n] [-nomail] [-noretry] [-nowait] \[-nowarn] [-R] [ -r rpass] [-u usrnm] loc_flnm [MSS_flnm][-nowarn] [-R] [ -r rpass] [-u usrnm] loc_flnm [MSS_flnm]

• mswritemswrite• mswrite [-C cos[ [-c cmnt] [-eof] [-f fmt] [-l n] [-m n] \mswrite [-C cos[ [-c cmnt] [-eof] [-f fmt] [-l n] [-m n] \

[-nomail] [-noretry] [-nowait] [-P proj_num] [-r pass] \[-nomail] [-noretry] [-nowait] [-P proj_num] [-r pass] \[-t n] [-v vmn] [ -w pass] loc_flnm [MSS_flnm] [-t n] [-v vmn] [ -w pass] loc_flnm [MSS_flnm]

• msrcp• msrcp [-a[sync] -cl[ass] cos -n[oreplace] -pe[riod] n \

-pr[oject] proj_num -rpwd rpass -wpwd wpass -R -V[ersion]] \ source_file [source_file...] target

Scientific Computing Division

MSS Access from User Applications

• f90 -L /usr/local/lib -l mss test.f• msread - retrieve file from mass store and make it local

• mswrite - write local file to mass store

• msrecall - wait for specific async read/write to complete

• mswait - check status of all queued async reads/writes to complete

• mserror - get error messages for last failed read/write

Scientific Computing Division

The End