tg 06 srb intro tutorial
TRANSCRIPT
-
8/14/2019 TG 06 SRB intro tutorial
1/26
By: Roman Olschanowsky
An Introduction to the
-
8/14/2019 TG 06 SRB intro tutorial
2/26
Outline
SDSC and History of SRB
Example Project
Introduction to SRB
Discussion on SRB basics
SRB Clients
Overview of a Data Grid
Infrastructure
Topology
Teragrid Demo
How to use your TG SRB account
How to access Digital Data Collections
-
8/14/2019 TG 06 SRB intro tutorial
3/26
ArchivalSystems
18 PB
15.6 TF
DataStarIBM Power4
4.4 TF
TeraGrid LinuxCluster (IA64)
1.4 PB
Storage AreaNetwork Disk Sun F15K
DiskServer
NetworkingVisualization
Storageand Compute
Resources
Human infrastructure:Experienced multi-
disciplinary staff support a
broad spectrum of nationalscience, engineering
and technology projects
Blue Gene/L
(Due 12/04)
2.8/5.7 TF
www.sdsc.edu
http://www.sdsc.edu/http://www.sdsc.edu/ -
8/14/2019 TG 06 SRB intro tutorial
4/26
Sites Using the SRBCiteSeer, Penn State
City Univ. of New YorkGeospatial Environment, UCSD
Drexel UniversityEOSDIS Distributed Active, NASA Goddard
Georgia TechKentucky State Libraries & Archives
Library of Congress
Los Alamos National Lab
NASA Ames
NASA Goddard Space Flight Center
NCSA Grid Computing
NIH (NCI Center for Bioinformatics)
Penn State University
Pittsburgh Supercomputing Center
Purdue University. Indiana
Stanford University
TACC, University of Texas
Texas A & M
UC Santa Cruz
UCLAUCSD Neuroscience
University of Maryland
University of Michigan, CAC department
University of New Mexico
University of Washington
University of Wisconsin
USC
Yale University
Academia Sinica, Taiwan
ASCC, Computing Centre, Taiwan
Australian National UniversityBedford Oceanography,Canada
Bioinformatics Institute, SingaporeCSIRO, Australia
Data Storage Institute, SingaporeEGEE, French National CenterGeoForschungsZentrum, Germany
James Cook University, Australia
KEK High Energy Physics, JapanMax Planck Institute, NetherlandsParallab, Norway
South Australian Advanced ComputingUIB (Parallab) , NorwayUniversity of Amsterdam
University of Cambridge, AstronomyUniversity of Cambridge, e-Science
University of EdinburghUniversity of Genoa, Italy
University of Hong KongUnivrsity of Manchester
University of OsloUniversity of SouthamptonYork Univ (UK)
-
8/14/2019 TG 06 SRB intro tutorial
5/26
SDSC SRB Projects (60 million, .5 PB )
Digital Libraries
UCB, Umich, UCSB, Stanford,CDL
NSF NSDL - UCAR / DLESE
NASA Information Power Grid
Astronomy
National Virtual Observatory
2MASS Project (2 Micron All Sky Survey)
Particle Physics
Particle Physics Data Grid (DOE)
GriPhyN
SLAC Synchrotron Data Repository
Medicine
Digital Embryo (NLM)
Earth Systems Sciences
ESIPS
LTER
Persistent Archives
NARA
LOC
Neuro Science & Molecular Science
TeleScience/NCMIR, BIRN
SLAC, AfCS,
-
8/14/2019 TG 06 SRB intro tutorial
6/26
Storage Resource Broker (SRB)
A distributed file system (Data
Grid)
Client-Server, Server-Server
architecture.
Abstracts physical
SRB provides the ability to
transparently share data acrossremote sites.
Heterogeneous Resources
Single sign on
Sin le lo ical file hierarchy
-
8/14/2019 TG 06 SRB intro tutorial
7/26
What we are familiar with
-
8/14/2019 TG 06 SRB intro tutorial
8/26
What we are not familiar with, yet
-
8/14/2019 TG 06 SRB intro tutorial
9/26
How do the file systems differ?
Logical Abstraction
Folders are NOT physicalFiles do NOT inherit physical location
Everything is potentially distributed
Access Control
Permissions are NOT rwxrwxrwxPermissions ARE on a object by objectbasis
Groups and permissions ARE more similarto NTFS
Domains
Geographical / logical grouping of users
Namespace scalability: john@harvardjohn@mit
Also doubles as groups
-
8/14/2019 TG 06 SRB intro tutorial
10/26
Interfaces to theStorage Resource Broker
inQ Windows Client
Scommands UNIX, DOS Command line
Client
Jargon Java API and GUIcomponents
mySRB Web Client
Matrix WSDL, Data Grid
WorkflowsC, C++ C and C++ API
Python Python API
Perl Perl API
-
8/14/2019 TG 06 SRB intro tutorial
11/26
Common Scommands (75 total)
Sinit
Senv
Spwd
Sls
Scd
Sget
SputSsh
Scp
Smv (logical)
Sphymove (physical)
Srm
Smkdir
Srmdir
Serror
Schmod
Sexit
-
8/14/2019 TG 06 SRB intro tutorial
12/26
mySRB
-
8/14/2019 TG 06 SRB intro tutorial
13/26
BIRN Portal (perl based)
-
8/14/2019 TG 06 SRB intro tutorial
14/26
NEEScentral Portal (php based)
-
8/14/2019 TG 06 SRB intro tutorial
15/26
Biomedical Informatics ResearchNetwork (BIRN)
Major collaboration with SDSC, several of theprojects Co-Investigators and Co-PIs are at
SDSC.
BIRNs purpose is to provide its consortiumof neuroscience laboratories the ability to
share, compute, and collaborate.
The Storage Resource Broker provides the
ability to transparently share data across
remote sites.
-
8/14/2019 TG 06 SRB intro tutorial
16/26
The BIRN SRB Data Grid
-
8/14/2019 TG 06 SRB intro tutorial
17/26
Doing this Manually
-
8/14/2019 TG 06 SRB intro tutorial
18/26
The BIRN Data Grid
-
8/14/2019 TG 06 SRB intro tutorial
19/26
The grid is in the details
-
8/14/2019 TG 06 SRB intro tutorial
20/26
File Replication
Sls
/home/Demo/SRB-Tutorial/files-2:
Doc.txt
Sls -l
/home/Demo/SRB-Tutorial/files-2:
romanoly 0 z-ucsd-ncmir-nas1 15 2003-07-09-05.15 Doc.txt
romanoly 1 z-jhu-cis-nas0 15 2003-07-09-05.16 Doc.txt
romanoly 2 z-stanford-lucas-nas 15 2003-07-09-05.16 Doc.txt
romanoly 3 z-umn-cmrr-nas0 15 2003-07-09-
05.16 Doc.txtromanoly 4 z-uci-bic-nas0 15 2003-07-
09-05.17 Doc.txt
-
8/14/2019 TG 06 SRB intro tutorial
21/26
Teragrid SRB
All Teragrid accounts are given a SDSC SRB Teragrid
accountThe username is the same as your SDSC UNIX account
name
Your SRB domain is teragrid
You must register your DN string with SDSCs grid-
mapfile or request a SRB password to activate your SRB
account. Instructions to do so are here:http://www.sdsc.edu/user_services/srb/account_form.html
Your ~/.srb/.MdasEnv file OR env variables
mdasCollectionHome '/home/.teragrid'
mdasDomainName 'teragrid'
srbUser ''
#AUTH_SCHEME 'ENCRYPT1'
AUTH_SCHEME 'GSI_AUTH'
srbHost 'srb.sdsc.edu'
srbPort '7321'
defaultResource 'sfs-tape-tgd'
SERVER_DN '/C=US/O=NPACI/OU=SDSC/CN=Storage ResourceBroker/USERID=srb'
http://www.sdsc.edu/user_services/srb/account_form.htmlhttp://www.sdsc.edu/user_services/srb/account_form.html -
8/14/2019 TG 06 SRB intro tutorial
22/26
Scommand Features
Command line interface -> SCRIPTING
Available for all of the most popular UNIX
flavors and DOS
S-commands are the most flexible and powerful
of the clientsThey are the fastest, and most reliable
They are multithreaded for big gains in data
flow
They are great for scripts, perl wrappers,
batch jobs, etc
Installed man pages via man [Scommand]
man Sput
-
8/14/2019 TG 06 SRB intro tutorial
23/26
Scommand Notes
Shelp
Gives list of commands with brief summary
[Scommand] gives usage info
(usually) or try h flag
Sinit establishes a session
Senv displays connection information
Spwd display current working directory
Sexit ends session
-
8/14/2019 TG 06 SRB intro tutorial
24/26
Some Public SRB Collections
Southern California Earthquake Center
/home/public.teragrid/SCEC
Two Micron All Sky Survey
/home/public.teragrid/2MASS
The Palomar Digital Sky Survey
/home/public.teragrid/DPOSS
-
8/14/2019 TG 06 SRB intro tutorial
25/26
Watch me do a SRB demo
-
8/14/2019 TG 06 SRB intro tutorial
26/26
Thanks!
SRB handles large data and provides theability to share and collaborate on
distributed heterogeneous resources.
Questions?
Teragrid SRB userguide:
http://www.sdsc.edu/user_services/srb
SRB website:
www.sdsc.edu/srb
SRB email:
srb@sdsc edu
http://www.sdsc.edu/user_services/srbhttp://www.sdsc.edu/srbmailto:[email protected]:[email protected]://www.sdsc.edu/srbhttp://www.sdsc.edu/user_services/srb