status of the tier-1 facilities at fermilab michael ernst fermilab february 10, 2004

Status of the Tier-1 Facilities Status of the Tier-1 Facilities at Fermilabat Fermilab

Michael ErnstMichael ErnstFermilabFermilab

February 10, 2004February 10, 2004

GDB Meeting February 10, 2004 2Michael Ernst Fermilab

Goals: Goals: Aggressive prototyping, early roll out, track external “practices”Aggressive prototyping, early roll out, track external “practices”

Approach: Approach: “Rolling Prototypes”: evolution of the facility and data systems“Rolling Prototypes”: evolution of the facility and data systems

Test stands for various hardware components and (fabric related software components) -- this allows to sample emerging technologies with small risks Setup of a test(bed) system out of next-generation components -- always keeping a well-understood and functional production system intact Deployment of a production-quality facility --- comprised of well-defined components with well-defined interfaces that can be upgraded component-wise with a well-defined mechanism for changing the components to minimize risks This matches to general strategy of “rolling replacements” and thereby upgrading facility capacity making use of Moore’s law

GGeenneerraall

AApppprrooaacchh ttoo

UUFF

WWBBSS


Projects at the Tier-1 FacilityProjects at the Tier-1 Facility

WBS 1.1.1.1 CPU and Disk Storage Technology EvaluationWBS 1.1.1.1 CPU and Disk Storage Technology Evaluation Jointly with CD/CSS Department Primarily NAS products Development of an integrated benchmarking suite

WBS 1.1.1.2 Storage Element WBS 1.1.1.2 Storage Element Jointly with CD/CCF Department and LCG/CERN-IT Working on a set of tools for DC04 to efficiently move files out of

CERN to the Tier-1 center(s) and between the Tier-1 and the Tier-2 centers

Advanced feature set, like resource reservation, prioritization using transfer queues (traffic shaping), third party transfer etc.

In US-CMS we foresee using SRM to interface the FNAL MSS to disk storage systems at the Tier-2 centers

All Tier-2 centers will have dCache installations (we prepared RPMs) All Tier-2 centers will have SRM interfaces

LCG-2 includes SRM V1 compatible Storage Element based on dCache and Castor



WBS 1.1.1.3 Configuration and Installation ManagementWBS 1.1.1.3 Configuration and Installation Management Need to supply centralized OS distribution for Tier-1/2 compute

nodes, capable of dynamic cluster partitioning and eases cluster administration and configuration stability

NPACI ROCKS was chosen after evaluation (02/11 – 03/04) Allows to define compute node configurations for different production

environments (making dynamic farm partitioning simple and fast) Reinstallation is the fundamental management and configuration tool.

Have augmented ROCKS with Yum and for dynamic updates Has shown clear advantages over FNAL based Linux distribution and

Systemimager tools

WBS 1.1.1.5 NetworkingWBS 1.1.1.5 Networking Ultra Scale Networking (n * Gbps) High-throughput and robust data movers Invited to use DataTag resources (Starlight/CERN link @ 10

Gbps)



WBS 1.1.2.3 Prototype for interactive and batch analysisWBS 1.1.2.3 Prototype for interactive and batch analysis Analysis activities scheduled in 2004 require increased services

and capabilities


Projects at the Tier-1 FacilityProjects at the Tier-1 FacilityData Streaming ProjectUS-CMS S&C has proposed joint project with CERN IT to developexpertise in wide area data streaming in preparation of LHC experiments.Interested to evaluate extending the physics reach of the detector byhaving dedicated analysis streams that cannot be reconstructed at CERNdue to limited resources, but might be reconstructed at the Tier1 centers

• Show feasibility of selecting, transferring, archiving streams of events in real time between Tier-0 center at CERN and remote Tier-1 facilties

• Serves as pilot for additional collaborative projects between CERN and the US

• Objectives include reliable data transport and network optimization archiving and storage event selection and identification

• Will build on and leverage effort from storage, networking and computing sub-projects in progress at Fermilab, I.e. the Fermilab Network Laboratory Project the Storage Resource Management (SRM) Collaboration the US-CMS Core Application Software Project


Worker NodeWorker Node

Worker NodeWorker Node

10 Disk Servers

Raw Data Streaming Evaluation ProjectRaw Data Streaming Evaluation Project

100 Dual XeonCompute Nodes

Filtering

Streaming

4 9940Tape Drives

4 9940Tape Drives

30 MB/s average

100 MB/s peak

AuxiliaryStreaming 5 Disk Servers

Fermilab CERN



Data Streaming Project (contd.)We are currently in procurement phase of the project. Equipment at CERN for event selection and deep buffering Equipment at FNAL for input buffers and tape throughput

We plan by late spring to demonstrate real time data streaming from CERN Start with end-to-end SRM transfers using improved network More advanced selection and transfer techniques that exercise the

CMS software should follow


Tier-1 Facilities CapacityTier-1 Facilities Capacity

OOffffssiittee CCoonnttrriibbuuttiioonn ttoo CCMMSS CCoommppuuttiinngg CCaappaacciittyy iinn 22000033

• Total: 2000 kSI2000

• US Tier-1 Facility: 200 kSI2000 (10%)

• Installed production capacity until end of 2003: 74 kSI2000


In order to be ready for the Data Challenge the User FacilitiesIn order to be ready for the Data Challenge the User Facilities

Need to increase the US-CMS Capacity Increase the processing and storage capacities at the Tier1

and Tier2 centers

Need to increase the services offered R&D effort to increase the facilities’ efficiency Networking, data serving and transfer, etc.

Need to improve the automation of simulation processing and later event

reconstruction Improvements and extensions to the Distributed Processing

Environment (DPE)– Increase in scale and robustness– Testing middleware and components– Establishing grid services

PPrreeppaarriinngg tthhee

FFaacciilliittiieess


Predicted resources for CMSPredicted resources for CMS The US Tier1 Center represents 10% of all the offsite resources Likely we need more boxes due to economic downturn and lack of market pressure

Significant Upgrade of Tier-1 FacilitySignificant Upgrade of Tier-1 FacilityScaling up the Tier-1 equipment Following the baseline plan In preparation for DC04 CPU, storage, data access

2002 2002 20032003

CPU [kSI2000]CPU [kSI2000]

Storage [TB]Storage [TB]

Throughput [MB/s]Throughput [MB/s]

7575

1010

200 200

200200

3434

700700

kSI2

000.

mo

nth

s


LSI E5600

NAS Head

Cisco 3750

NAS/dCacheServer

1

20

Cisco 3750

NAS/dCacheServer

1

20

Cisco 3750

NAS/dCacheServer

1

16

…

Production Cell

FC Switch (64 Ports)

FNAL Site Net

link trunking(4* GigE)

Infortrend A16F

~ 4 TB/ea

dCache PoolsUser Area

Enstore

Production Cell Analysis Cell

32 Gbps SwitchInterconnect

Crossbar for Disk I/O

WAN

20/16Dual Xeon(2.4 GHz)Systems/Cell

Cell based Computing Architecture


FY03 UpgradeFY03 Upgrade

76 Worker Nodes (Dual 1 U Servers)76 Worker Nodes (Dual 1 U Servers) 120 CPUs for Production (99 kSI2000) 32 CPUs for Analysis (26 kSI2000) SMC X5DPA w/Intel 7501, 533 MHz Frontside Bus 2 * 2.4 GHz XEON Processors 2GB Memory 2 Disk Drives (40 + 120 GB) 10/100/1000 Ethernet on-board

17 Server Nodes17 Server Nodes Intel SE7501 Dual XEON E-ATX Server Board PCI-X 64 bit/100 MHz 2 * 2.8 GHz XEON Processors 2/4 GB Memory


StorageStorageWe have been looking for a highly available, reasonably performing We have been looking for a highly available, reasonably performing

storage solution to provide analysis space for usersstorage solution to provide analysis space for users• Users now make requests that cannot be easily accommodated by

our existing facilities• Have looked at a number of Network Attached Storage (NAS) and

Storage Area Network (SAN) systemsInvestigated three possible storage solutionsInvestigated three possible storage solutions

• A NAS system for common space and user space and 3ware RAID systems for large scale inexpensive storage

• A NAS system for common space and user space and separate storage modules attached to servers

• A large SAN system that combines the common and user space with the large scale storage

Have decided to go for a mix, bridging the solutions at the level of the network attached data servers

NAS/SAN solution (IBRIX) for common/user space Inexpensive Fiber Channel attached RAID systems for dCache pools This is a very flexible and robust solution


FY03 Upgrade - StorageFY03 Upgrade - Storage

5 inexpensive Infortrend A16F Serial ATA Disk Arrays (~20TB)5 inexpensive Infortrend A16F Serial ATA Disk Arrays (~20TB) 2 * 2Gbps FC Interfaces 16 * 300GB SATA Drives RAID level 0, 1, 1 (0+1), 3, 5, 10, 30, 50 (multiple RAID selections) 2 redundant hot-swappable Power Supplies

1 expensive, high performance LSI LOGIC E5600 (~3.5 TB) 1 expensive, high performance LSI LOGIC E5600 (~3.5 TB) 2 RAID Controllers

8 * 2Gbps FC Host connections (800MB/s max.) 4 * 2Gbps FC Drive connections (800MB/s max.) Number of Drives supported 14 – 224 (32.7 TB max.) max. 795MB/s (or 53.200 I/O ops/s sustained)

NAS Heads based on IBRIX Cluster File System running on DellNAS Heads based on IBRIX Cluster File System running on Dell2650 Servers exporting file systems via NFS2650 Servers exporting file systems via NFS

Evaluated beta software Production release arrived in late December


FY03 Upgrade - NetworkingFY03 Upgrade - Networking

5 Cisco 3750 Modular Switches5 Cisco 3750 Modular Switches 24 Ethernet 10/100/1000 ports and 4 Fiber uplinks Stackable (up to 9 units, 32-Gbps high-speed stacking bus) Single image, full Cisco Operating System

QLOGIC SANbox 2-64 Fiber Channel SwitchQLOGIC SANbox 2-64 Fiber Channel Switch 64 2Gbps Fiber Channel Ports (expandable by 8 port blades) Up to 256 Gbps (full duplex) fabric aggregate bandwidth (412MB/s P-to-P)


Proposal for Tier-1 Center FY04 ProcurementsProposal for Tier-1 Center FY04 Procurements

Production SystemsAdditional resources required to meet obligations for official CMSsimulated event production and data challenges• CMS cellular production architecture still compatible with CMS

Computing Model. Allows to increase number of cells to increase available resources

• Since power and cooling at Fermilab’s FCC is limited we propose a staged procurement 2 cells (40 worker nodes, 2 4TB RAID systems, 2 servers, 2 GigE

switches) to be procured soon and installed at FCC 2 cells to be procured toward the end of FY04 and installed at HDCF


Proposal for Tier-1 Center FY04 ProcurementsProposal for Tier-1 Center FY04 Procurements

Analysis SystemsAdditional resources are required to meet the needs of the US physics

community for analysis• CMS cellular production architecture still compatible with CMS

Computing Model. Allows to increase number of cells to increase available resources

• Since the system will provide interactive services we propose to install it at Fermilab’s FCC (UPS/Generator backed power) 1 cell (20 worker nodes, 1 4TB RAID systems, 2 servers, 1 GigE

switches)


Offsite data transfer requirements have consistently outpaced available bandwidthOffsite data transfer requirements have consistently outpaced available bandwidth Upgrade by

ESnet to OC12 (12/02) becoming heavily utilized at times

FNAL planning to obtain an optical network connection to the premier optical network switching center FNAL planning to obtain an optical network connection to the premier optical network switching center on the North on the North American American continent – continent – StarLight in StarLight in Chicago, Chicago, enables enables network network research research and holds and holds promise forpromise for

Handling peak production loads for times when production demand exceeds what

ESnet can supply. Acting as a

backup in case the ESnet link is unavailable

Potential on a single fiber pairPotential on a single fiber pair Wavelength

Division Multiplexing (WDM) for multiple independent data links

Allows to configure bandwidth to provide a mix of immediate service upgrades as

well as validation of non-traditional network architectures

Immediate benefit to production bulk data transfers, a test bed for high performance network Immediate benefit to production bulk data transfers, a test bed for high performance network investigationinvestigations and s and scalability scalability into the area into the area of LHC of LHC operations operations

IImmpprroovviinngg

oouurr sseerrvviicceess::

NNeettwwoorrkkiinngg


AAllll FFNNAALL ooffff--ssiittee ttrraaffffiicc ccaarrrriieedd bbyy EESSnneett lliinnkk

EESSnneett CChhiiccaaggoo PPooiinntt ooff PPrreesseennccee ((PPooPP)) hhaass 11GGbb//ss SSttaarrLLiigghhtt lliinnkk

Peering with CERN, Surfnet, CAnet there

Also peering with Abilene there (for now)

EESSnneett ppeeeerrss wwiitthh ootthheerr nneettwwoorrkkss aatt ootthheerr ppllaacceess

CCuurrrreenntt

OOffff--ssiittee

NNeettwwoorrkkiinngg


DDaarrkk ffiibbeerr iiss aann aalltteerrnnaattee ppaatthh ttoo SSttaarrLLiigghhtt--ccoonnnneecctteedd nneettwwoorrkkss

AAllssoo aann aalltteerrnnaattee ppaatthh bbaacckk iinnttoo EESSnneett

PPrrooppoosseedd

NNeettwwoorrkk

CCoonnffiigguurraattiioonn


Need to study, understand and optimize the end-to-end network packet flow in the CMS wide area network production environmentNeed to study, understand and optimize the end-to-end network packet flow in the CMS wide area network production environment With GridFTP we are taking advantage of empirically discovered mechanism of

striping data transfers across a set of parallel TCP connections Crucial for application development is a sound understanding of the underlying

mechanisms that explain how parallel TCP connections improve aggregate

throughput Joint project with CD/CCF on High Performance WAN data transfer optimization

- Develop, validate and document a model that incorporates network stack and OS

information and its analysis to improve network performance

- Develop the tools to avoid making CMS distributed applications network-aware

and to force application developers to deeply understand wide variety of

monitoring tools and tuning methods Aiming at Installations/Evaluations for Integration with Production Environment at

Fermilab, CERN and Tier-2 sites Datatag offered CERN/StarLight link

EEnndd--ttoo--EEnndd

PPeerrffoorrmmaannccee//

NNeettwwoorrkk

PPeerrffoorrmmaannccee aanndd

PPrreeddiiccttiioonn


We’ve made good progress with dCache to replace some essential functionality formerly provided by the Objectivity AMSWe’ve made good progress with dCache to replace some essential functionality formerly provided by the Objectivity AMS dCache is a disk caching system jointly developed by DESY and Fermilab

as a front end for Mass Storage Systems We are using it as a way to serve data from disks attached to worker nodes

in demanding applications like simulation with pile-up. Applicat ions access the data in dCache space over a POSIX compliant interface.

From the user perspective the dCache namespace (/pnfs) looks like any other cross mounted f ile system. URL s tyle address ing is used wherever pnfs cannot be mounted

Essential set of features for load balancing and error recovery dCache can replicate f iles between pools if the load on a server is above a

configurable threshold I f a server fails in an ins tallation with tape backend, dCache will restore files on

remaining pools . The applicat ion stalls until data is available.

IImmpprroovviinngg

DDaattaa

SSeerrvveerr

SSeerrvviiccee


High Throughput Pile-up Simulation

High Throughput and New Functionalities through dCache and dCap


Simulation of CMS detector is difficultSimulation of CMS detector is difficult

There are 17 interactions per crossing on average There are 25ns between crossings The previous 5 crossing and the following 3 influence the detector response. Each simulated signal event requires 170 minimum bias events To simulate new minimum bias events would take about 90 minutes A large sample is created and recycled

- The sample is sufficiently large, it doesn’t usually fit on local disk

It is about 70MB per event These events are randomly sampled, so it is taxing on the minimum bias

servers and the network

DDaattaa IInntteennssiivvee

AApppplliiccaattiioonnss


DDccaacchhee ttoo

SSeerrvvee

PPiillee--uupp

Pile-up events are stored across the pools

As the load on a pool node increases the dCache Server can trigger a pool-to-pool replication of a file and balance the load

Pool Node

Pool Node

Pool Node

Pool Node

Pool Node

Pool Node

Pool Node

Pool Node

Pool Node

PNFSWorker Node

libpdcap.so

Local Disk

writeAllDigis

Dcache Server

URLPOOLCat.


PPeerrffoorrmmaannccee iiss ffaaiirrllyy ffllaatt aaccrroossss tthhee nnooddeess aanndd tthhee ppeerrffoorrmmaannccee iiss ggoooodd

TThhee ppeerrffoorrmmaannccee iinn tthhee ppiillee--uupp aapppplliiccaattiioonn iiss ssuuffffiicciieenntt tthhaatt tthhee aannaallyyssiiss aapppplliiccaattiioonn sshhoouulldd bbee wweellll sseerrvveedd..

DDaattaa

RRaattee IInnttoo tthhee

AApppplliiccaattiioonn


Found scaling issues with Data Transfer, Storage Management,

Grid Job Scheduling (Condor-G), Process Management at 200

CPUs Found scaling issues in the client architectures Issues with writing out too much information into common

areas

Current methods of data management are insufficient for a large scale distributed production systemCurrent methods of data management are insufficient for a large scale distributed production system Output is written using globus-url-copy from headnode This prevents nodes from needing external network access,

but stresses headnode At a minimum we need the ability to queue transfers.

- Currently transfers commence as soon as the jobs are finished. A real data management system is needed

WWhhaatt

wwee lleeaarrnneedd


As we increase the amount of data generated by our automated production system, and as we prepare for analysis applications, we need to improve the data management tools deployed.As we increase the amount of data generated by our automated production system, and as we prepare for analysis applications, we need to improve the data management tools deployed.

CMS has taken a two pronged approachCMS has taken a two pronged approach CMS has adopted the Storage Resource Broker (SRB), developed in part by

PPDG, to handle our data transfers during Pre-Challenge Production SRB is a working solution, which is well supported and has a global catalogue The SRB architecture currently has some limitations

US-CMS/FNAL-CD, CERN and the LCG are jointly developing the Storage Element (SE) which will be used to provide some data management services

Based on the Storage Resource Manager (SRM) and Replica Location services (RLS)

GGrriidd

DDaattaa

SSttoorraaggee

aanndd

DDaattaa

AAcccceessss


TThhee SSttoorraaggee EElleemmeenntt ddeevveellooppmmeenntt iiss bbaasseedd oonn SSRRMM..

SRM provides a uniform interface to diverse and distributed

physical storage devices (MSS, Disks, Data Caching services, etc.)

BBaasseedd

oonn

SSttoorraaggee

RReessoouurrccee

MMaannaaggeerr

Storage System Abstraction

OSM Enstore TSM

SRM

HPSS

dCache Cache

HSM

SRM SRM

Virtual Storage LayerSRM

SRM ClientStore/Retrieve

Pre-staging, Space allocation, PinningTransfer Protocol Negotiation

Data Transfer (e.g. GridFTP)Client SystemSystem System

Castor


Pool Manager

I/O Door Nodes

SRMGFtp dCap (K)Ftp(Krb5,ssl)

Http Admin

File Name SpaceDatabase

NFS Serverpnfs

File Name SpaceProvider

OSM Enstore TSM

Admin DoorsdCache ComponentsdCache Components

Pool Nodes

HSMs


Advantages of using SRMsAdvantages of using SRMsProvides uniform Grid access to heterogeneous Mass Storage Systems Provides uniform Grid access to heterogeneous Mass Storage Systems Synchronization between storage resourcesSynchronization between storage resources

Pinning file, releasing files (responsibility ?) Allocating space dynamically on as “needed basis”

Insulate clients from storage and network system failuresInsulate clients from storage and network system failures Transient MSS failure Network failures Interruption of large file transfers Data Corruption (data in transit; calculate, store, compare checksum)

Facilitate file sharingFacilitate file sharing Eliminate unnecessary file transfers

Support “streaming model”Support “streaming model” Use space allocation policies by SRMs: no reservations in advance needed Use explicit release by client for reuse of space

Control number of concurrent file transfers (queuing and traffic shaping)Control number of concurrent file transfers (queuing and traffic shaping) From/to MSS – avoid flooding Head/Gateway Node, MSS and thrashing From/to network – avoid flooding and packet loss


Data Transfer Service and Cataloguing

Fermilab (Tier1) Caltech (Tier2) San Diego (Tier2) Florida (Tier2) CERN (Tier0)

UCSD

Florida

Caltech

Fermilab

CERN

SRB

SRM/GridFTP

dCache/Enstore

Castor

SRM

/Grid

FTP


In the US we have identified manpower to develop and integrate the Storage ElementIn the US we have identified manpower to develop and integrate the Storage Element SRM development US CMS is driving further development of the V2 specifications The local area data serving tools (dCap, rfio, ROOTD, nfs) are

stable and well debugged The Replica Location Service (RLS) that creates all the data

cataloguing services are still in prototypes Several implementations not all of which are compatible Still lacking higher level data management function, e.g the

interface of the catalogue services to the applications is still in

the development phase Still working on how the SE communicates with POOL and the

applications

SSttaattuuss

ooff

SSEE

DDeevveellooppmmeenntt

PPrroojjeecctt


After DC04 is completed, CMS will enter a year of intense analysis activity preparing for the Physics TDR. US-CMS needs to increase the analysis capability of the Tier1 in order to encourage more US Physicists to use themAfter DC04 is completed, CMS will enter a year of intense analysis activity preparing for the Physics TDR. US-CMS needs to increase the analysis capability of the Tier1 in order to encourage more US Physicists to use them A lot of the preparations overlap with DC04 preparations Data serving Data management Software distributions Some are extensions of DC04 preparations Need some simple VO management for DC04, but the multiple user environment of

analysis requires more services Extensions of the production tools to allow custom user simulation and distributed

analysis applications Some are new efforts Load balancing for interactive users and the user analysis environment

AAnnaallyyssiiss

PPrreeppaarraattiioonn


The first virtual organization infrastructure will be deployed for DC04 pre-production, but there are only a few production users and the application is predictable The first virtual organization infrastructure will be deployed for DC04 pre-production, but there are only a few production users and the application is predictable and organizedand organized

We don’t worry production users will do something malicious or foolish

The analysis environment is much more complicated.The analysis environment is much more complicated. Many more users with diverse applications, abilities, and access patterns

The VO Project is working with US-ATLAS to developed the infrastructure for authenticating and authorizing usersThe VO Project is working with US-ATLAS to developed the infrastructure for authenticating and authorizing users First prototypes concentrate on authentication Need to satisfy experiment wide and local site policies Authorization at the level of individual resources is necessary and it soon

couples to auditing and usage policies Requires a new project that will be launched in early 2004, will address– Rights to perform tasks and access services– Currently handled by mapping individuals to local UNIX accounts on clusters, works poorly

for Grids

– Need better solution for grid users to handle file access, queue access and other services

VVOO

MMaannaaggeemmeenntt

status of the tier-1 facilities at fermilab michael ernst fermilab february 10, 2004

Documents

facility wbs

michael ernst fermilab

gdb meeting

uf wbs slide

facility capacity

gbps slide

productionquality facility

castor slide