srm basic spec lbnl proposal
DESCRIPTION
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N. SRM Basic Spec LBNL Proposal. - PowerPoint PPT PresentationTRANSCRIPT
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N
SRM Basic Spec
LBNL Proposal
Arie Shoshani, Alex Sim, Junmin Gu, Kurt StockingerScientific Data Management Group
Lawrence Berkeley National Laboratory
Summary
• LBNL proposal of SRM Basic specification based on v.1.1• Original v.1.1 methods/arguments in BLUE
Proposed modifications in Green for essential additions with + to the v.1.1 Skyblue for optional additions with ++ to the v.1.1 Red for subtractions from the v.1.1 Purple for renaming and implied updates of the v.1.1
with =>
• Notations:{} … tuple() … list (ordered collection)[] … set (unordered collection)_ … underline as mandatory (as opposed to optional)::= … definition operator| … choice operator (exclusive or)// … commentitalic … recommended
Basically
• From v.2.1.1 or in addition to v.1.1 No space management
o Automatic minimum directory management
o Ls, removeFileso No mkdir, etc.
No permission managemento Somewhat automatic
Some new functionso srmGetSystemInfo // ConfigInfoo Ping
Relationship to v.1.1 and v.2.1.1 (1)
v.1.1 This version v.2.1.1
requestID = integer requestID=string requestToken
=string
requestStatus Separated into RequestSummary and ReturnRequestStatus. FileStatuses from v.1.1 is separated out.
Split into RequestSummary and RequestFileStatus as the response to the request.
FileMetaData Enhanced FileMetaData
MetaDataPathDetail
Relationship to v.1.1 and v.2.1.1 (2)
v.1.1 This version v.2.1.1
RequestFileStatus RequestFileStatus Many different request status. E.g. GetRequestFileStatus, PutRequestFileStatus, etc.
StatusCode Enhanced and
concrete
Explanation added
extensive
automatic storagetype FileStorageType desired to be added
fileStorageType added
Relationship to v.1.1 and v.2.1.1 (3)
v.1.1 This version v.2.1.1
Get Enhanced srmPrepareToGetAdded more parameters
Put Enhanced srmPrepareToPutAdded more parameters
Copy Enhanced srmCopyAdded more parameters
Relationship to v.1.1 and v.2.1.1 (4)
v.1.1 This version v.2.1.1
MkPermanent srmChangeFileStorageType srmChangeFileStorageType
Pin Automatic when file is ready automatic
UnPin srmRelease, or automatic when file lifetime is expired
srmRelease or automatic
Additional pinning to extend lifetime ?
srmExtendedFileLifeTime added
srmExtendedFileLifeTime added
Relationship to v.1.1 and v.2.1.1 (5)
v.1.1 This version v.2.1.1
getRequestStatus srmGetRequestStatus to return RequestSummary
Separated into request type. E.g. srmStatusOf{Get,Put,Copy}Request
Together in getRequestStatus
srmGetRequestFilesStatus added to return detailed information on each file
Separated into request type. E.g. srmStatusOf{Get,Put,Copy}Request
getFileMetadata srmLS with ability to return single directory meta information
srmLS with extensive
Options Enhanced
directory management
Relationship to v.1.1 and v.2.1.1 (6)
v.1.1 This version v.2.1.1
getProtocols srmGetTransferProtocols none
getEstGetTime Absorbed into srmGetRequestStatus
Or srmStatusOf{Get,Put,Copy}Request
Absorbed into srmStatusOfGetRequest
getEstPutTime Absorbed into srmGetRequestStatus
Or srmStatusOf{Get,Put,Copy}Request
Absorbed into
srmStatusOfPutRequest
Relationship to v.1.1 and v.2.1.1 (7)
v.1.1 This version v.2.1.1
setFileStatus srmPutDone Enhanced into srmChangeFileStorageType
Or srmPutDone
AdvisoryDelete srmRemoveFiles ? srmRemoveFiles
Automatic space management
Automatic Space Management
Space Management
Permission Management
minimal Directory management
GET
• In: SURLs, ProtocolsAuthorizationID,
DesiredFileLifeTime,FileStorageType
Out: RequestStatusSummary
RequestID
• Basic assumptions: UserID
• Get()
• In: SURLs, Protocols
Out: RequestStatus
• srmPrepareToGet()
• In:authorizationID,arrayOfGetFileRequest,arrayOfTransferProtocols, userRequestDescription, storageSystemInfo,
totalRetryTime
• Out: requestToken,
returnStatus,
arrayOfFileStatus
RequestStatusSummary
• RequestIDType (Get, Put, … ) State ErrorMessage FileStatusesRetryDeltaTimeSubmitTimeStartTimeFinishTimeEstimatedTimeToStart
• RequestIDRequestTypeStatusCode // RequestErrorCodeStatusExplanation // ErrorDetailsSubmitTimeStartTimeFinishTimeEstimatedTimeToStartTotalFilesInRequestNumberOfQueuedFilesInRequestNumberOfFinishedFilesInRequestNumberOfProcessingFilesInRequest
• requestToken,requestType,
totalFilesInThisRequest,
numOfQueuedRequests,
numOfFinishedRequests,
numOfProgressingRequests,isSuspended
PUT
• Put()
• In: Src_file_names Dest_file_names, Sizes WantPermanent Protocols
• Out: RequestStatus
• In: {Dest_SURLSize }[]
FileStorageType Protocols AuthorizationID OverwriteMode DesiredLifeTime
• Out: RequestStatusSummary RequestID
• srmPrepareToPut()
• In: authorizationIDarrayOfFileRequest,
arrayOfTransferProtocols,
userRequestDescription,
overwriteOption,
storageSystemInfo,
totalRetryTime
• Out: requestToken,
returnStatus,
arrayOfPutFileStatus
COPY
• Copy()
• In: Source_SURLs, Dest_SURLs
• Out: RequestStatus
• In: { Source_SURL,Dest_SURL }[]
AuthorizationID
DesiredLifeTime
FileStorageType
OverwriteModeOut: RequestStatusSummary RequestID
• srmCopy()
• In: authorizationID,
arrayOfCopyFileRequest,
userRequestDescription,
overwriteOption,
removeSourceFilesstorageSystemInfo,
totalRetryTime
• Out: requestToken,
returnStatus,
arrayOfCopyFileStatus
MkPermanent
• MkPermanent• IN: SURLs• OUT: ReturnStatus
• srmChangeFileStorageType()
• In: authorizationID,
arrayOfPath,
desiredStorageType,
spaceToken
• Out: returnStatus,
arrayOfFileStatus
PIN
• Obsolete
• Automatic when file is ready
• In: TURLs
• Out: RequestStatus
UnPin
• UnPin()
• In: RequestIDTURLs
• Out: RequestStatus
• UnPin => Release
• In: RequestIDTURLs | SURLsAuthorizationID
• Out: { TURL | SURL
RequestStatusCode Explanation } []
• srmReleaseFile()
IN: requestToken authorizationID,
siteURLs,
keepSpace
• Out: returnStatus,
arrayOfFileStatus
getRequestStatus
• In: RequestID
• Out: RequestStatus => RequestStatusSummary
• For details: getRequestFilesStatus (previous getRequestStatus which used to include RequestFileStatus and FileMetaData)
• getRequestStatus()
• In: RequestID
• Out: RequestStatus
• srmStatus(Get,Put,Copy)Request
• IN: requestToken,
authorizationID
arrayOfFromSURLs,
• Out: returnStatus,
arrayOfFileSta
getFileMetaData
• getFileMetaData
• In: SURLs
• Out: FileMetaData
• getFileMetaData => LS
• In: SURLsAuthorizationIDIsAllLevelRecursiveListOffsetListCount
infoOnly
• Out: FileMetaData
• srmLs()
• IN: authorizationID,
path,
fileStorageType,
fullDetailedList,
allLevelRecursive,
numOfLevels,
offset,
count
• Out: MetaDataPathDetail[]
returnStatus
FileMetaData
• SURLSizeOwnerGroupPermModeChecksumTypeChecksumValueIsPermanent IsPinned IsCached
• SURLSizeOwnerGroupPermModeChecksumTypeChecksumValueFileStorageTypeStatusCodeStatusCodeLifetimeAssignedLifetimeLeftCreatedTimeLastModifiedTimeFileTypeOriginalSURLSubDirectoryPTR // dirEntries
• path,
status,
size, // 0 if dir
ownerPermission,
userPermission,
groupPermission,
otherPermission
createdAtTime,
lastModificationTime,
owner,
fileStorageType,
fileType, // Directory or FilelifetimeAssigned,
lifetimeLeft,
checkSumType,
checkSumValue,
originalSURL, MetaDataSubPathDetail[]
getProtocols
• In: none
• Out: list of transfer protocols
• None in v.2.1
• Let’s keep this or put into the new srmSystemInfo or srmConfigInfo
getEst[Get/PUT]Time
• Make it Obsolete
• getEstGetTime: • Out- RequestStatus
In- SURLs, Protocols
• getEstPutTime: • Out- RequestStatus
In- Src_file_name, dest_files_names, sizes, wantPermanent,Protocols
• getEstimationTime
• In: Source_SURLDest_SURLSize
AuthorizationID
TransferProtocols
FileStorageType
• Out: RequestEstimation
• None• Similar one is
getStatusRequest() after request is submitted
AdvisoryDelete
• Obsolete
• AdvisoryDelete:
• Out- noneIn- SURLs
• srmRemoveFiles
• In: SURLs authorizationID
• Out: ReturnRequestStatus
• srmRemoveFiles
• IN: requestToken,
authorizationID,
siteURLs
• Out: returnStatus,
arrayOfFileStatus
GetRequestFilesStatus
• In: RequestIDOut: RequestFileStatus []
• In: RequestIDListOffsetListCount
• Out: RequestFileStatus []
• srmStatusOf{Get, Put, Copy}Request
• IN: requestToken,
authorizationID
arrayOfFromSURLs,
• Out: returnStatus,
arrayOfFileStatus
RequestFileStatus
• SURL SizeState FileldTURLOwner, Group, PermModeCheckSumType, CheckSumValueIsPinned, IsPermanent QueueOrderEstSecondsToStart SourceFilenameDestFilenameIsCached
• Source_SURLDest_SURLSizeStatusCode Explanation TURLCheckSumType, CheckSumValueFileStorageTypeEstWaitingTimeOnQueueEstProcessingTimeSourceFilenameDestFilenameRemainingLifetime
• fromSURLInfo,fileSize,
• status,
• estimatedWaitTimeOnQueue,
• estimatedProcessingTime,
• transferURL
• remainingPinTime
• siteURL,
• toSURL,
Ping
• In: none
• Out: alive_response
GetConfig
• In: flag• Out: { value |
list of supported functions}
• e.g. GET, PUT, COPY, DYNAMIC_MKDIR, PUSH_MODE, PULL_MODE,Default_space = 5GB/user, etc.
New types
• FileStorageType VOLATILE, DURABLE, PERMANENT
• FileType FILE, DIRECTORY, LINK
• OverwriteMode NEVER, ALWAYS, WHEN_DIFFERENT
• RequestType GET, PUT, COPY
• StatusCode // RequestStatusCode, FileStatusCode,
RequestErrorCode, FileErrorCode Explanation: human-readable strings // ErrorDetails
• RequestID : from int to string
Conclusion
• Based on SRM v.1.1 and many of v.2.1.1• Included much of Olof’s issues and suggestions to
v.1.1• Operational specification would be bound to Web
Services.