srm & se jens g jensen wp5 atf, december 2002. collaborators rutherford appleton (atlas...
TRANSCRIPT
![Page 1: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/1.jpg)
SRM & SE
Jens G Jensen
WP5
ATF, December 2002
![Page 2: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/2.jpg)
Collaborators
• Rutherford Appleton (ATLAS datastore)
• CERN (CASTOR)
• Fermilab
• Jefferson Lab
• Lawrence Berkeley– Americans have: ENSTORE, SAM, …
• Also present: WP2
For an overview of storage systems see D5.1
![Page 3: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/3.jpg)
File pinning & replication
LocalSRM
RemoteSRM
RC
filefile(replica)
• Client fetches a file using srmGet(LFN,SURL)• Local SRM gets a replica identified by LFN.• Issue: LFN is optional
Other issues:
What are defautl ACLs on replicas?
Does the local SRM have to check ACLs from the remote SRM?
![Page 4: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/4.jpg)
RCs• SRM must be able to interoperate with
several different RCs
• Getting SURLs (RC SRM) is easy, but:– Does SRM need reverse lookup (SURL
LFN)?
– How to notify RC that a file has been deleted?• Add hooks to SRM?
![Page 5: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/5.jpg)
SRM file lifetime overviewFiles are:• Volatile (e.g. pinned files)
– Files get deleted by SE when they time out– Files created as volatile are never written to tape
• Durable (e.g. permanent but not in final location)– Files can be deleted only by administrator or owner– Files still have a lifetime – owner and/or admin
notified when lifetime expires
• Permanent (e.g. your important experiment)– Files can be deleted only by administrator or owner
![Page 6: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/6.jpg)
New lifetime stuff: space
• Each user has space with a per-user (probably) quota for each type of space
• Users must reserve space before they can reserve for writing or reading
• Users are charged for the type and amount of space they reserve
• Space can be– Volatile – reservations not guaranteed beyond a minimum– Durable – notify user if file didn’t move, space wasn’t
released, etc – if user doesn’t react, notify admin– Permanent
• SRM Proposal: space lifetime is encoded in the filepath!!! (JLAB has this)
Proposal: space has a lifetime as well.
![Page 7: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/7.jpg)
Storage Semantics
Volatile Durable Permanent
![Page 8: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/8.jpg)
copyToSpace
• Users can “give away” files in their own space to another user– First step: the user who owns the files gives them
away– Second step: the other user takes over the files– Done without copying in the SRM, but quotas are
adjusted accordingly
• Alternatively, let users “give” files to other users without deleting them from their own space– copy-on-write
![Page 9: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/9.jpg)
SRM possibly wildly inaccurate status
GetPULL
PutPUSH
CopyFile
Lifetime
Space
LifetimeGSIFTP
SESE 1.0
TB 2.0
SE 1.0
TB 2.0
SE 1.1
TB 2.1
SE 1.1
TB 2.1
SE 1.x
x>=1
SE 1.0
TB 2.0
Fermi OK OK OK OKonly vol
-- OK
JLAB OK OK OK OK -- OK
![Page 10: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/10.jpg)
Comparing SRM v2.{0,1} to SE 1.0
• Get, Put: supported– Get in PULL mode (same as SRM)– Put in PUSH mode (same as SRM)
• File lifetime: will be easy to support– Currently (pre-1.0) all files are permanent…
• Space management – currently supported as volatile, best-effort only (i.e. very little support)
• Information providers: supported soon• Directory stuff : mostly supported (not rmdir, no
recursion)• Support for arbitrarily long LFNs: OK
Need to have API for more detailed comparison…
![Page 11: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/11.jpg)
Need for POSIX style access
• RFIO (CERN, and other sites with HPSS)
• DCAP (FermiLab)
• Globus XIO?
• file:
Pick one, and call it a standard?
![Page 12: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/12.jpg)
Security• Americans expect to use CAS
– Possibly need for interim solution until CAS is sufficiently mature?
• Need for “weak” ftp?– Users “delegate” username/password to SRM
• Europeans expect to use VOMS– SE solution until VOMS is sufficiently mature:
GACL (SE will always support GACL)– SE can use CAS as well but support not planned
• Use GSI-HTTPS for now – common ground essential for interoperability
Question: are CAS and VOMS interchangeable?
![Page 13: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/13.jpg)
ACLs• How to manage ACLs on replicas?
– Replicas are read-only; or– Replicas are writable by whoever created the replica– SRM may have to contact the SRM containing the
master to pick up changes to the ACL
• SRM proposal: default is world readable for volatile space, owner access only for permanent
• SE proposal– Have default ACL per user and/or per directory
![Page 14: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/14.jpg)
Web services
• Fermi uses GLUE but plans to switch to Apache/Tomcat/Axis.
• “Secret” SRM v1.0 WSDL available…
• New WSDL spec for v2.1 specification in preparation (Fermi)
• SOAP interoperability issues, particularly with parameters more complicated than strings and ints – e.g. (WSDL) structs.
• Need to be OGSI compliant…
![Page 15: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/15.jpg)
Schema
• Agree that GLUE is important
• More effort needed?
![Page 16: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/16.jpg)
How to ensure that users behave?
• Good behaviour == close file && release pin• Make them pay… Charge per space reservations
and/or size of files in user’s reservation?– CASTOR charges per MB on tape
• Count number of expired pins (in durable and permanent only!?)
• What to do with files that are opened and never closed?– Can be detected sometimes (local files, NFS?,
/grid?)– CASTOR currently does nothing
![Page 17: SRM & SE Jens G Jensen WP5 ATF, December 2002. Collaborators Rutherford Appleton (ATLAS datastore) CERN (CASTOR) Fermilab Jefferson Lab Lawrence Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022071705/56649f585503460f94c7d91a/html5/thumbnails/17.jpg)
Collaborations…
• Agree standard protocols, accept each others certificates, etc.
Testing Interoperability
• Sharing code is seen as less important– Occasional licence conflict– People would rather have more independent
implementations