us cms software and computing the project plan matthias kasemann fermilab fnal oversight panel

October 23, 2000 USCMS S&C Project Matthias Kasemann 1

US CMS Software and Computing The Project Plan Matthias Kasemann

Fermilab

FNAL Oversight Panel October 23, 2000


Talk OutlineTalk Outline

Overview of the US CMS Project Management Plan Scope of the US CMS S&C project Organization of the US CMS S&C Project Relation to Int’l CMS Status and Future Evolution of the PMP

Current status of the US CMS S&C Project


LHC Software & Computing ChallengesLHC Software & Computing Challenges

Geographical dispersion: of people and resources Complexity: the detector and the LHC environment Scale: Petabytes per year of data

1800Physicists (360 from US)

150 Institutes 32 Countries

Major challenges associated with:Major challenges associated with: Coordinated Use of Distributed computing resources Coordinated Use of Distributed computing resources Remote software development and physics analysisRemote software development and physics analysis Communication and collaboration at a distanceCommunication and collaboration at a distance

R&D: New Forms of Distributed Systems: Data-GridsR&D: New Forms of Distributed Systems: Data-Grids


US and Worldwide Data GridUS and Worldwide Data Grid

Tier2 Center

Online System

Offline Farm,CERN Computer

Center

FNAL CenterFrance Center

Italy Center UK Center

InstituteInstituteInstituteInstitute ~0.25TIPS

Workstations

~100 MBytes/sec

~2.4 Gbits/sec

100 - 1000

Mbits/sec

Bunch crossing per 25 nsecs.100 triggers per secondEvent is ~1 MByte in size

Physicists work on analysis “channels”.Each institute has ~10 physicists working on one or more channels

Physics data cache

~PBytes/sec

~0.6 - 2.5 Gbits/sec + Air Freight

Tier2 CenterTier2 CenterTier2 Center~622 Mbits/sec

Tier 0 +1

Tier 1

Tier 3

Tier 4

Tier2 Center Tier 2

Experiment


The Goal of the “S&C Project”The Goal of the “S&C Project”

To provide the software and computing resources needed to enable US physicists to fully participate in the physics program of CMS

Allow US physicists to play key roles and exert an appropriate level of leadership in all stages of computing related activities …

From software infrastructure to reconstruction to extraction of physics results

From their home institutions, as well as at CERN

This capability should extend to physicists workingat their home institutions


US-CMS Software and Computing ProjectUS-CMS Software and Computing Project

Aug 1998 First presentation to DOE/NSF (Newman + Taylor) 12/98 - 6/00 Project Management Plan (PMP) 11/98, 5/99 Favorable DOE/NSF “Peer” Reviews;

Funding for Software Engineers 11/98 - 12/99 Biannual Reports and Discussion at JOG 7/99 - Acting Software and Computing Board; L1/L2 PMs 12/99 Submission of Draft PMP to DOE/NSF 1/00 WBS, Draft Funding Profile for 2000-2006

Presented at DOE/NSF Peer Review 3/00 Start Up Funding for UF Subproject 5/17/00 GriPhyN ITR Proposal (Grid IT R&D) 5/20/00 SC Organization Approved and PMP by US CMS CB 5/30/00 DOE/NSF Proposed Funding Profile 6/12/00 US CMS Response, Discussion at JOG 6/25/00 Finalize updated PMP, sent to collaboration 6/29/00 Quarterly Progress Report at DOE 6/00 - 11/00 Update Requirements and Costs; Refine WBS 7/00 - 10/00 Plan Tier2 Center Prototype Startup 11/14/00 Project Baselining Review by DOE/NSF 11/00 L1 PM Starts; Appoints L2 PMs (CAS, UF)


Outline of Software and Computing PlanOutline of Software and Computing Plan Goals of the Project Project Organization Upper Level Project Management Interrelationship with Other Entities Evolution of the US CMS Software and Computing Project

Newest Version is October 20, 2000 (separated Project Plan Management Plan from WBS, milestones and budget)

Additional Documents describe:High Level Milestones, Work Breakdown Structure, and

Budget for the User Facilities SubprojectHigh Level Milestones, Work Breakdown Structure, and

Budget for the Core Applications SubprojectHigh Level Milestones, Work Breakdown Structure, and

Budget for Overall ProjectAppendix:

CMS Software Milestones and WBS


Participants/Key PlayersParticipants/Key Players

CERN International CMS US CMS US Funding agencies US Universities and National Labs US Physicists


Key Management Entities Key Management Entities Level 1 Project Manager + deputy Level 2 Project Manager for Core Applications Software Level 2 Project Manager for User Facilities The Advisory Software and Computing Board -- ASCB The Fermilab Director or Designee advised by the Project

Management Group The Fermilab Computing Division US funding agencies and Joint Oversight Group


US Funding Agencies and

Fermilab (CMS Host Institution)

Fiscal Authority

Policy & reportingSoftware andComputing Project

Project Manager Liaison

Core Applications

Software

User Facilities

Support

US CMS AdvisorySoftware

and Computing Board

(USASCB)

ReconstructionDetector Software

andPhysics Groups

Organization of US CMS S&C ProjectOrganization of US CMS S&C Project


US CMS Software and Computing Scope US CMS Software and Computing Scope

The US CMS Software and Computing Project Core Application Software User Facilities

Tier 1 and Tier 2 regional Centers

Networking support CERN--US/Fermilab Internal to US

Reconstruction and Detector SoftwarePhysics Analysis

These are closely related but are dealt with outside of this particular project as an “extended project” or as “related activities”

Project Scope


CAS SubprojectCAS Subproject

In addition to developing software, this subproject will also provide expert programming personnel to assist the physicists in developing reconstruction and physics analysis programs by serving as mentors, reviewers, advisers and, where appropriate as software tool-writers. This will ensure that the software produced will be easy to integrate into the whole system, will be efficient in its use of hardware resources, and will be maintainable and adaptable for the full life of the CMS data analysis.

To support the design, development, modeling optimization and commissioning of software related to detectors being constructed by US CMSTo provide its share of the framework and infrastructure software required to support data analysis and simulation for CMSFor remote collaborative tools to enable the distributed model of computing that permits members of US CMS to carry out analysis whether they are at home in the US or visiting or resident at CERNTo satisfy any specialized needs required to carry out data analysis activities of interest to members of US CMS


UF SubprojectUF Subproject

The goal of the User Facilities Subproject is to provide the enabling infrastructure of software and computing that will allow US physicists to fully participate in the physics program of CMS.

To this end the subproject will acquire, develop, install, integrate, commission and operate the hardware and software for the facilities required to support the development and data analysis activities of USCMS.

This subproject will include a major ‘Tier1’ regional computing center at Fermilab to support US physicists working on CMS. It is appropriately sized to support this community which comprises 20% of the full CMS collaboration.Tier 2 Centers are part of the User Facilities Subproject


Tier 2 CentersTier 2 CentersTier 2s will partner with a Tier 1 which will provide

support and liaison with CERN.US CMS Tier 2 Centers (or the CMS parts of multiple

customer centers) appear as Level 3 WBS items under the UF project, and the UF Level 2 manager has specific responsibilities with respect to them

Tier 1 interfaces to the Tier 2’s in the USThe Tier 2 CMS representative is a L3 manager of

the US S&C project


Joint OversightGroup

FermilabDirector

Or Designee

Construction ProjectProject ManagerTechnical Director

Advisory Board

HCAL

ECAL

TRIDAS

EMU

Tracker

CP

PO

PHYSICSANALYSIS

TaskOUTREACH

HCAL IB

ECAL IB

TRIDAS IB

EMU IB

Tracker IB

Software andComputing Project

Project Manager

CoreApplications

Software

UserFacilities

CollaborationBoard

Advisory SoftwareAnd ComputingBoard (ASCB)

Reconstructionand DetectorSoftware Task

ProjectManagement

Group

Fiscal Authority

Policy & reportingLiaison

Support

Organization of US CMS ProjectsOrganization of US CMS Projects


Role of the ASCBRole of the ASCB

Helps in Development of the Project Plan Advises on Scientific and Technical Policy Advises on appointments Provides Continuous Project Input and Feedback

throughout the life of the construction project

The US CMS Software and Computing Board provides crucial input and feedback for the US CMS Software and Computing project. It advises the Level 1 Project Manager and provides liaison to US CMS.


Composition of ASCBComposition of ASCB Specifies elected and Ex Officio members

six members elected at large from the US CMS collaboration; the US CMS Physics Coordinator (also elected); the Chair of the US CMS Collaboration Board (ex-officio) the Head of the Fermilab Computing Division (ex-officio) the CMS Project Manager for Core Software + Computing (ex-

officio) the Project Manager of the US CMS Construction Project (ex-

officio). Level 1 Project Manager of the US CMS Software and Computing

Project (ex-officio) Two Level 2 Project Managers of the US CMS Software and

Computing Project (ex-officio)The 7 elected members will choose a chairperson from among

the 6 members who were elected at large Term is 2 years, 3 positions elected every year


Relation to CMSRelation to CMS

To create a strong linkage between the US CMS Software and Computing Project and the overall CMS Project, the CMS L2 manager will be an ex officio member of the USASCB. The USASCB will have the responsibility for providing liaison between the US CMS Computing Project and the CMS Software and Computing Board. The US CMS Software and Computing Project Level 1 Project Manager will act as liaison to the CMS Software and Computing Technical Board.


US-CMS Sub-System Links to International CMSUS-CMS Sub-System Links to International CMS

Collaboration Board

HCAL PM

ECAL PM

TRIDAS PM

EMU PM

Tracker PM

Common Proj. PM

HCAL PM

ECAL PM

TRIDAS PM

Muon PM

Tracker PM

Note: Subsystem Institution and Finance Boards are omitted for clarity

SW/Comp. PM

SW/Comp. PM

Magnet PM

Management Board

Steering Committee

US-CMS CMS


““Extended” Lines of CommunicationExtended” Lines of Communication

CMS SCBCMS SCTB

US S&C L1PM US ASCB Collaboration-elected members, chair of US CMS CB, Physics Coordinator Construction Project Technical Director FNAL CD -- head of CD Technical -- L1, L2 PMs CMS -- S&C Project ManagerJoint Oversight

Group

Fermilab Director or Designee

Software andComputing Project

Project Manager

Project Management

Group Fermilab Director or Designee -- aka FNAL Deputy Director and Assoc. Director for Research

Technical -- L1, L2 PMs Collaboration -- chair of USASCB and

chair of US CMS CB Construction Project Technical

Director Head of FNAL CD


External Review CommitteeExternal Review Committee

The chair of the PMG will establish a standing external review committee that will periodically examine and evaluate all aspects of the US CMS Software and Computing Project.


Formal Agreements with CMSFormal Agreements with CMSThe US CMS collaboration will be contributing to

CMS computing in a variety of ways, each of which will have an appropriate formal mechanism for establishing milestones, deliverables, and specifications.

Levels of support for production activities, including those required to support the design, testing, simulation, and commissioning of the detector should be supported by MOUs negotiated with CMS by the L1PM with input from the US ASCB and with the approval of the PMG and funding agencies.


Formal Agreements with CMS (cont’d)Formal Agreements with CMS (cont’d)

The software development that directly relates to the international CMS effort should be developed as part of the CMS software and computing plan and approved, presumably as part of the project plan for the US CMS Software and Computing Project, by the PMG and the funding agencies.

Software efforts specifically in support of US physicists or intended to solve particular problems specific to the US, should be developed as part of the project plan with substantial input from US ASCB and approved by the PMG and, if required, by the funding agencies.


The Project Organization: StatusThe Project Organization: Status

Advisory Software and Computing Board (ASCB) Chair I. Gaines (FNAL) for 1 year Elected Members P. Avery (Florida) for 2 years

S. Dasu (Wisconsin) for 1 yearS. Eno, (Maryland) for 1 yearS. Kunori (Maryland) for 2 yearsD. Stickland (Princeton) for 2 years

Physics Coordinator J. Branson (UCSD) for 2 years Chair of US CMS CB H. Newman (Caltech), ex-officio Construction Project Manager D. Green (FNAL), ex-officio Int’l CMS S&C Coordinator M. Pimia (CERN) ex-officio Head of FNAL/CD M. Kasemann ex-officio Acting L1 PM M. Kasemann ex-officio Acting L2 PM – CAS L. Taylor ex-officio Acting L2 PM – UF V. O’Dell ex-officioLevel 1 Project Manager L. Bauerdick (DESY/ZEUS -> FNAL) Appointment Confirmed by JOG 6/00 Starts November 2000


US CMS S&C Subprojects: StatusUS CMS S&C Subprojects: Status Core Application Software Subproject

Resource-loaded WBS for CMS and US CMS Task- Oriented Requirements + US Support US part: 9 (2000) - 13 (2004) FTEs of

software engineering User Facilities Subproject

Implement R&D and Prototype Systems: 1999-2003 Preproduction ODBMS and Event-distribution

systems Simulated Event service for PRS and ORCA teams

Implement Production Systems in 2004-2006 Replenish and Upgrade from 2006 - Staff:

Tier 1: 35 FTEs by 2006, partly leveraged from FNAL/CD Tier 2: 5 1.5 FTEs = 7.5 FTEs

Caltech,NEU,

Princeton,UC Davis

+ FNAL

FNAL +Tier 2 sites


US CMS S&C: Data GridUS CMS S&C: Data GridDeploy computing resources as hierarchical grid

Tier 0 Central laboratory computing resources (CERN) Tier 1 National center (Fermilab / BNL) Tier 2 Regional computing center (Universities) Tier 3 University group computing resources Tier 4 Individual workstation/CPU

“Data Grid” to reflect the predominant role of data in the distributed analysis; Concept developed by Caltech, U. Florida and FNAL

This concept and “Tiers” has been adopted throughout Europeby the EU Grid Project

We have to start implementation of Tier 1 and Tier2 R&D systems now to:

Support physics and HLT studies Support software development Perform R&D on distributed data analysis


US CMS Tier2 planningUS CMS Tier2 planning FY 2000/1 Build basic services; HLT milestones

1-2 prototype-Tier 2 Studies with ORCA FY 2001 Initial Grid system File replication service

work with Tier 1 Multi-site cached file access FY 2002 Second set of Tier2 CMS Data Challenges

centers S&C TDR and Physics TDR FY 2003/4 Tier 2 centers 5%-Scale Data Challenge +

at last set of sites PhysicsTDR;production data grid test

FY 2004/5 Production-quality 20% Production Grid System CMS Mock Data Challenge

FY 2005/6 Full distributed systemFinal Production Shakedown

R&D systems: leverage existing resources at Universities Funding for Tier 2’s to come mostly from NSF initiatives


US CMS Tier 2 CentersUS CMS Tier 2 Centers Discussed at US CMS Collaboration Board, May 20, 2000

Agreed: start deployment of 1 prototype Tier 2 center Query sent to each US CMS institute:

Interest and plans to participate in Tier2 R&D now? Who can work on it?

Result: Ready to started “now”: Caltech + UC Davis + UCSD9/00

Florida, 4/01Univ. of Iowa + Iowa State ~4/01

Start later for production: Boston, Maryland, MinnesotaWisconsin

Next steps: Consolidate Tier2 plans for

R&D on distributed data analysispossibility to contribute to Monte Carlo productionPossible leverage from Universities


First Prototype Tier 2 in CaliforniaFirst Prototype Tier 2 in California Caltech + UC San Diego + UC Davis

Hardware Plan + Cost:80 Dual CPU + Disk Linux Nodes $ 200 k [40 in Caltech, 40 in UCSD]Sun Data Server with RAID Array $ 30 k [partly from Caltech]Tape Library $ 20 k [in Caltech]LAN Switches $ 50 kCollaborative Infrastructure Upgrades $ 10 kInstallation and Infrastructure $ 30 kNet Connect to Abilene $ 0 kTape Media and Consumables $ 10 kStaff (Ops and System Support) $ 50 kGigabit data access at UC Davis $ 30 kTotal Estimated Cost (First Year) $ 430 kUCSD cost sharing $ -50 kUC Davis cost sharing $ -30 kCost $ 350 k

Status: Funding allocation request submitted to start first Tier 2 center in CaliforniaHardware is arriving at sites


PMP: Status/Summary - IPMP: Status/Summary - I WBS exists for the two major subprojects Resource estimates exist

Revised recently: With input from Int’l CMS planning during Hoffmann Review at CERN After first funding guidelines received from DOE in May

Milestones and timelines exist and are frequent and aggressive enough to drive the project and its oversight

Key management roles are identified ASCB elected in September 2000 L1 PM identified, starts November 2000

Key interfaces are worked out in a manner consistent with the collaborative nature of CMS and US CMS and with their

“scientific culture” Key interfaces are worked out with funding agencies and CMS A draft of a full Project Management Plan exists

We separated the “stable” (project organization and oversight) aspects of the plan from the more “changeable” aspects (funding dependent). We created two documents:

Project Management and Work Breakdown Structure Schedules, Milestones, and Budgets (finishing now)


PMP: Status/Summary - IIPMP: Status/Summary - II The plan draws strength and exploits the synergy

between US Universities and Fermilab Software Professionals and physicists

It takes advantage of and contributes to key developments in the US in information technology

Drive towards a high speed network infrastructure Development of ever better network software and

applications such as grid computing concepts It takes advantage through the Tier2 centers of the

significant strengths of US universities in the area of computer science and information technology

These are key elements to its eventual success