a test framework for movement of mc data using sam & srm interface
DESCRIPTION
A Test Framework for Movement of MC Data using SAM & SRM Interface. 2 nd March, 2008 CDF Offline Meeting. Overview. Test Framework Composition of files Monitoring Parameters Time Plot Transfer Rate Cumulative Data Plot Query Map Conclusion & Next Step Wish List !. - PowerPoint PPT PresentationTRANSCRIPT
A Test Framework for Movement of MC Data
using SAM & SRM Interface
2nd March, 2008CDF Offline Meeting
2
Overview
Test FrameworkComposition of filesMonitoring Parameters
Time PlotTransfer RateCumulative Data PlotQuery Map
Conclusion & Next StepWish List !
Data Movement: Test Framework
Data from SAM, dccp
fcdflnx1cdfsam15
FNAL SRM V2srm://cmssrm.fnal.gov:8443
UCSD SRM V2srm://t2data2.t2.ucsd.edu:8443
scpsrmcp
srmcp
Composition of Files
4A SAM data file is assosciated with its meta data file.
Composition of Files: File Size
5
No. of Files No. of Files
Meta Data File Size in Bytes Data File Size in Mega Bytes
File
Siz
e
Meta data files are of the order of few Kilo bytes.Data files are the order of several Mega Bytes.
Monitoring Parameters: Transfer Time
6
No. of Files No. of Files
Meta Data Transfer Time in Sec Data Transfer Time in Sec
Tim
e Ta
ken
Meta data files (few Kilo bytes) take around 14 seconds.Data files takes (several Mega bytes) around 34 seconds.
Monitoring Parameters: Evolution of files with time
7
Time Time
Meta Data File Size in Bytes Data File Size in Mega Bytes
File
Siz
e
A dip in the right hand side plot (18:32 – 19:02) is being balanced by copying meta data files during this period.
Monitoring Parameters: Transfer Rate
8
No. of Files No. of Files
Meta Data Tr. Rate in Bytes/sec Data File Tr. Rate in MBs/sec
Tran
sfer
Rat
e
Transfer rate for larger size files is much higher than smaller size files.
Monitoring Parameters: Cumulative Flow of Data
9
File
Siz
e
Time
More than ~30 Giga Bytes/hr of data has been transferred. More test is needed to confirm a robust transfer rate.
Monitoring Parameters: Query Map
10
Not easy to draw any conclusions from the above plot. But, it will be helpful to display quality data for multiple sites versus time. Each box represents a fixed duration worth of transfers.
Conclusion & Next StepConclusion:
Larger the file size, higher the efficiency of the framework.
Next Steps Different sizes of files will be used
For files of other than above different sizes, joining files with “tar” is one option. But, it is too much cpu consuming.
In the next offline meeting, we will show plots on monitoring parameters for trans Atlantic movement of data. 11
DataSet Each File Size (in MB)
1. atop48 500
2. bhel0d 737
3. b0dsdka 850
4. ftop1b 10505. bhmu0i 1350
Wish List ! Addition of more SRM managed storage element sites for
studying the performance of the framework. Compatibility of SAM with SRM version 2
Present version of SRM in SAM is 1 Latest version of SRM is 2. SRM version 2 is not backward compatible
with SRM version 1. Present version of SAM needs some development such that it would be
compatible with SRM version 2 A rough idea of the time frame ( under which the above issues
will be resolved) will be helpful in speedy development of this work !
12