A Test Framework for Movement of MC Data

using SAM & SRM Interface

Manoj K. Jha (INFN- Bologna)Antonio Cuomo, Donatella Lucchesi, Gabriele Compostella, Simone Pagan (INFN – Padova)

Doug Benjamin (Duke University)Robert Illingworth (Fermilab)

8th April, 2008CDF Offline Meeting

2

Overview

Test FrameworkComposition of filesMonitoring Parameters

Time PlotTransfer RateCumulative Data Plot

Conclusion & Next StepIssues

Data Movement: Test Framework

cdfsam15

FNAL SRM V2srm://cmssrm.fnal.gov:8443

UCSD SRM V2srm://t2data2.t2.ucsd.edu:8443

srmcp

srmcp

A process is running on “cdfsam15” which create dummy files of random sizes. The random number generator is being biased for creating files of larger sizes (> 1 GB).

We are using WAN for transfer of files using srmcp. The no. of channels is 50 at UCSD SRM end.

Composition of Files

Type File Size (fsize)

A fsize <= 10 MB

B 10 MB < fsize <= 100 MB

C 100 MB < fsize <= 1GB

D 1GB < fsize <= 2GB

E 2GB < fsize <= 3GB

F fsize > 3GB

4

The dummy files generated in sizes from 1 MB to 4GB.Number in the pie chart indicates the number of files generated corresponding to that category from 3rd to 8th April, 2008.

Monitoring Parameters: Transfer Time

5Initializations time of SRM is roughly of the order of 10 sec.

Monitoring Parameters: Transfer Rate

6

Transfer rate increases with increase in file size.Maximum achievable transfer rate is around 60 MB/secUser analysis jobs will fall in category B while MC jobs in C & D

B

D

C

Monitoring Parameters: Transfer Time and Rate

Comparison of transfer time for diff. categories of file.

Comparison of transfer rate for diff. categories of file.

Monitoring Parameters: Cumulative Flow of Data/hr

8

File S

ize

Time

Transfer rate of 25 GB/hr is achievable using WAN. Transfer rate of 2 TB/hr is achievable using “FermiLight Path Circuits”. For more information, please visit http://indico.fnal.gov/getFile.py/access?contribId=8&sessionId=19&resId=0&materialId=0&confId=1037

Monitoring Parameters: Cumulative flow of data/day

9

Around 600 GB/day of data can be transferred using WAN. For data transfer of higher rate, we would have to consider other options like “FermiLight Path Circuits”.

Conclusion & Next StepConclusion:

SRM initialization time is of the order of 10 sec. Date transfer rate increases with file size. Maximum transfer rate achievable is around 60 MB/sec Transfer rate of 25 GB/hr and 600GB/day is achievable using WAN.

Next Steps Study the monitoring parameters for

Trans Atlantic movement of data. Data movement between worker nodes and SRM As a function of various number of requests arriving at SRM

Write a CDFNote explaining the implementation of framewrok within the CAF

10

Issues ! Authentication:

Presently, the user’s proxy is being generated on robotic head node. Users cannot take the proxy outside from the head node. What should be the mechanism for accessing the files on SRM, since users cannot access his/her proxy credentials ?

Addition of more SRM managed storage element sites will help us in speedy development of this work.

Compatibility of SAM with SRM version 2 Present version of SRM in SAM is 1 Latest version of SRM is 2. SRM version 2 is not backward compatible

with SRM version 1. Present version of SAM needs some development such that it would be

compatible with SRM version 2

11

Acknowledgement !

We would like to thank Jon Bakken (from FNAL), Frank Wuerthwein and Abhishek Singh Rana (from UCSD)

for providing resources on SRMs !

12