tony doyle - university of glasgow 31 january 2005gridpp12 collaboration meeting gridpp status...

31 January 2005 GridPP12 Collaboration Meeting Tony Doyle - University of Glasgow

GridPP Status Report

Tony Doyle


Contents

• What was GridPP1?

• What is GridPP2?• Challenges abound• LCG

– Issues

• Deployment Status (9-28-30/1/05) – Tier-1/A, Tier-2,

NGS

• M/S/N• Middleware• Food chains• Applications • Dissemination• The UK mountain

climb• Summary


What was GridPP1?

• A team that built a working prototype grid of significant scale

> 2,000 (9,000) CPUs> 1,000 (5,000) TB of available storage> 1,000 (6,000) simultaneous jobs

• A complex project where 88% of the milestones were completed and all metrics were within specification

1 . 1 2 . 1 3 . 1 4 . 1 5 . 1 6 . 1 7 . 1

1 . 1 . 1 1 . 1 . 2 1 . 1 . 3 1 . 1 . 4 2 . 1 . 1 2 . 1 . 2 2 . 1 . 3 2 . 1 . 4 3 . 1 . 1 3 . 1 . 2 3 . 1 . 3 3 . 1 . 4 4 . 1 . 1 4 . 1 . 2 4 . 1 . 3 4 . 1 . 4 5 . 1 . 1 5 . 1 . 2 5 . 1 . 3 6 . 1 . 1 6 . 1 . 2 6 . 1 . 3 6 . 1 . 4 7 . 1 . 1 7 . 1 . 2 7 . 1 . 3 7 . 1 . 41 . 1 . 5 2 . 1 . 5 2 . 1 . 6 2 . 1 . 7 2 . 1 . 8 3 . 1 . 5 3 . 1 . 6 3 . 1 . 7 3 . 1 . 8 4 . 1 . 5 4 . 1 . 6 4 . 1 . 7 4 . 1 . 8 6 . 1 . 5

2 . 1 . 9 3 . 1 . 9 3 . 1 . 1 0 4 . 1 . 9

1 . 2 2 . 2 3 . 2 4 . 2 5 . 2 6 . 2 7 . 2 1 . 2 . 1 1 . 2 . 2 1 . 2 . 3 1 . 2 . 4 2 . 2 . 1 2 . 2 . 2 2 . 2 . 3 2 . 2 . 4 3 . 2 . 1 3 . 2 . 2 3 . 2 . 3 3 . 2 . 4 4 . 2 . 1 4 . 2 . 2 4 . 2 . 3 4 . 2 . 4 5 . 2 . 1 5 . 2 . 2 5 . 2 . 3 6 . 2 . 1 6 . 2 . 2 6 . 2 . 3 7 . 2 . 1 7 . 2 . 2 7 . 2 . 31 . 2 . 5 1 . 2 . 6 1 . 2 . 7 1 . 2 . 8 2 . 2 . 5 2 . 2 . 6 2 . 2 . 7 3 . 2 . 5 3 . 2 . 6 3 . 2 . 7 3 . 2 . 8 4 . 2 . 5 4 . 2 . 6 4 . 2 . 71 . 2 . 9 1 . 2 . 1 0 3 . 2 . 9

1 . 3 2 . 3 3 . 3 4 . 3 5 . 3 6 . 3 7 . 3

1 . 3 . 1 1 . 3 . 2 1 . 3 . 3 1 . 3 . 4 2 . 3 . 1 2 . 3 . 2 2 . 3 . 3 2 . 3 . 4 3 . 3 . 1 3 . 3 . 2 3 . 3 . 3 3 . 3 . 4 4 . 3 . 1 4 . 3 . 2 4 . 3 . 3 4 . 3 . 4 5 . 3 . 1 5 . 3 . 2 5 . 3 . 3 6 . 3 . 1 6 . 3 . 2 6 . 3 . 3 6 . 3 . 4 7 . 3 . 1 7 . 3 . 2 7 . 3 . 3 7 . 3 . 41 . 3 . 5 1 . 3 . 6 1 . 3 . 7 1 . 3 . 8 2 . 3 . 5 2 . 3 . 6 2 . 3 . 7 3 . 3 . 5 3 . 3 . 6 4 . 3 . 51 . 3 . 9 1 . 3 . 1 0 1 . 3 . 1 1

1 . 4 2 . 4 3 . 4 4 . 4 5 . 4 1 . 4 . 1 1 . 4 . 2 1 . 4 . 3 1 . 4 . 4 2 . 4 . 1 2 . 4 . 2 2 . 4 . 3 2 . 4 . 4 3 . 4 . 1 3 . 4 . 2 3 . 4 . 3 3 . 4 . 4 4 . 4 . 1 4 . 4 . 2 4 . 4 . 3 4 . 4 . 4 5 . 4 . 1 5 . 4 . 2 5 . 4 . 3 5 . 4 . 41 . 4 . 5 1 . 4 . 6 1 . 4 . 7 1 . 4 . 8 2 . 4 . 5 2 . 4 . 6 2 . 4 . 7 3 . 4 . 5 3 . 4 . 6 3 . 4 . 7 3 . 4 . 8 4 . 4 . 5 4 . 4 . 6 5 . 4 . 51 . 4 . 9 3 . 4 . 9 3 . 4 . 1 0 M e t r i c O K 1 . 1 . 1

M e t r i c n o t O K 1 . 1 . 1 1 . 5 2 . 5 3 . 5 4 . 5 T a s k c o m p le t e 1 . 1 . 1

1 . 5 . 1 1 . 5 . 2 1 . 5 . 3 1 . 5 . 4 2 . 5 . 1 2 . 5 . 2 2 . 5 . 3 2 . 5 . 4 3 . 5 . 1 3 . 5 . 2 3 . 5 . 3 3 . 5 . 4 4 . 5 . 1 4 . 5 . 2 4 . 5 . 3 4 . 5 . 4 T a s k o v e r d u e 1 . 1 . 11 . 5 . 5 1 . 5 . 6 1 . 5 . 7 1 . 5 . 8 2 . 5 . 5 2 . 5 . 6 2 . 5 . 7 3 . 5 . 5 3 . 5 . 6 3 . 5 . 7 6 0 d a y s 1 . 1 . 11 . 5 . 9 1 . 5 . 1 0 T a s k n o t d u e s o o n 1 . 1 . 1

N o t A c t i v e 1 . 1 . 1 2 . 6 3 . 6 4 . 6 N o T a s k o r m e t r i c

2 . 6 . 1 2 . 6 . 2 2 . 6 . 3 2 . 6 . 4 3 . 6 . 1 3 . 6 . 2 3 . 6 . 3 3 . 6 . 4 4 . 6 . 1 4 . 6 . 2 4 . 6 . 32 . 6 . 5 2 . 6 . 6 2 . 6 . 7 2 . 6 . 8 3 . 6 . 5 3 . 6 . 6 3 . 6 . 7 3 . 6 . 8 N a v ig a t e u p 2 . 6 . 9 3 . 6 . 9 3 . 6 . 1 0 3 . 6 . 1 1 3 . 6 . 1 2 N a v ig a t e d o w n

E x t e r n a l l i n k 2 . 7 3 . 7 L in k t o g o a l s

2 . 7 . 1 2 . 7 . 2 2 . 7 . 3 2 . 7 . 4 3 . 7 . 1 3 . 7 . 2 3 . 7 . 3 3 . 7 . 42 . 7 . 5 2 . 7 . 6 2 . 7 . 7 2 . 7 . 8 3 . 7 . 5 3 . 7 . 6

2 . 8 3 . 8 2 . 8 . 1 2 . 8 . 2 2 . 8 . 3 2 . 8 . 4 3 . 8 . 1 3 . 8 . 2 3 . 8 . 32 . 8 . 5

W P 8

1 2 3

D e p l o y m e n t

W P 4

W P 5

F a b r i c

T e c h n o l o g y

W P 6

D u e w i t h i n

A T L A S

G r i d P P G o a l

R e s o u r c e sI n t e r o p e r a b i l i t y D i s s e m i n a t i o n

T i e r - 1

T i e r - A

L H C b T i e r - 2

C E R N D a t a G r i d A p p l i c a t i o n s I n f r a s t r u c t u r e

W P 1

W P 2

W P 3

L C G C r e a t i o n

A p p l i c a t i o n s

W P 7

A T L A S / L H C b

C M S

B a B a r

C D F / D O

U K Q C D

O t h e r

D a t a C h a l l e n g e s

R o l l o u t

T e s t b e d

1 - J a n - 0 4S t a t u s D a t e

I n t . S t a n d a r d s

O p e n S o u r c e

W o r l d w i d e I n t e g r a t i o n

U K I n t e g r a t i o n

M o n i t o r i n g

D e v e l o p i n gE n g a g e m e n t

P a r t i c i p a t i o n

T o d e v e l o p a n d d e p l o y a l a r g e s c a l e s c i e n c e G r i di n t h e U K f o r t h e u s e o f t h e P a r t i c l e P h y s i c s c o m m u n i t y

P r e s e n t a t i o n D e p l o y m e n t

5 6 74

U p d a t e

C l e a r

A Success

“The achievement of

something desired, planned, or

attempted”


Executive Summary I

• “The GridPP1 Project is now complete: following 3 years of development, a prototype Grid has been established, meeting the requirements of the experiments and fully integrated with LCG, currently the World’s largest Grid. Starting from this strong foundation, a more complex project, GridPP2, has now started, with an extended team in the UK working towards a production Grid deployed for the benefit of all experiments by September 2007.”

• We achieved (almost exactly) what we stated we would do in building a prototype…


Executive Summary II

• “2004 was a pivotal year, marked by extraordinary and rapid change with respect to Grid deployment, in terms of scale and throughput. The scale of the Grid in the UK is more than 2000 CPUs and 1PB of disk storage (from a total of 9,000 CPUs and over 5PB internationally), providing a significant fraction of the total resources required by 2007. A peak load of almost 6,000 simultaneous jobs in August, with individual Resource Brokers able to handle up to 1,000 simultaneous jobs, gives confidence that the system should be able to scale up to the required 100,000 CPUs by 2007. A careful choice of sites leads to acceptable (>90%) throughput for the experiments, but the inherent complexity of the system is apparent and many operational improvements are required to establish and maintain a production Grid of the required scale. Numerous issues have been identified that are now being addressed as part of GridPP2 planning in order to establish the required resource for particle physics computing in the UK.”

• Most projects fail in going from prototype to production…

• There are many issues: methodical approach reqd.


What is GridPP2?

0. Production Grid

1. 1 2.1 3.1 4.1 5.1 6.1

1. 2 2.2 3.2 4.2 5.2 6.2

1. 3 2.3 3.3 4.3 6.3

1. 4 2.4 3.4 4.4 6.4

2.5 3.5 4.5

Navigate down External link Link to goals

2.6 3.6 4.6

Network

Management

& MonitoringInformation PhenoGrid

KnowledgeTransfer

32

Management

Grid Deployment Security CMS UKQCD

Engagement

Grid Technology Workload LHCb D0

Computing Fabric Data & Storage Ganga CDF Deployment

Grid Operations

1 6M/S/N LHC Apps

54

GridPP2 GoalTo develop and deploy a large scale production quality grid in the UK for the use of the Particle Physics community

Tier-A Tier-1 Tier-2 Deployment Middleware Support Experiment Support

Interoperability

ATLAS Dissemination

Management ExternalLCG

Planning

Applications Metadata

Non-LHC Apps

BaBar

SAMGrid

LHC Deployment Portal

Structures agreed and in place (except LCG phase-2)

•253 Milestones, 112 Monitoring Metrics at present.•Must deliver a “Production Grid”: robust, reliable, resilient, secure, stable service delivered to end-user applications. •The Collaboration aims to develop, deploy and operate a very large Production Grid in the UK for use by the worldwide particle physics community.


Must share data between thousands of scientists with multiple interestslink major (Tier-0 [Tier-1]) and minor (Tier-1 [Tier-2]) computer centresensure all data accessible anywhere, anytimegrow rapidly, yet remain reliable for more than a decadecope with different management policies of different centresensure data securitybe up and running routinely by 2007

What are the Grid challenges?



Data Management, Security and

Sharing

1. Software process2. Software efficiency3. Deployment

planning 4. Link centres

5. Share data

6. Manage data7. Install software8. Analyse data9. Accounting

10. Policies


What are the limits on Data?Advanced Areal Density Trends

M. Leonhardt 4-9-02

0.001

0.01

0.1

1

10

100

1000

10000

100000

1000000

1987 1992 1997 2002 2007 2012 2017 2022Year

Are

al D

ensi

ty (

Gb

/in2 )

Parallel Track Longitudinal Tape

Helical Tape

Magnetic Disk

Optical Disk

Superparamagnetic Effect

Probe Contact Area Viability

Atom Surface Density

Atom Level Storage

Probe

Volumetric

Optical

?

?

?

Tape Demos

?

Technical Progress

Technology BoundariesTechnical Progress

Technology Boundaries

Serpentine Longitudinal Tape

LHC era

1 PetaBit/in2 !!

1 Terabit/in2 !

Currently disk capacity doubles every year (or so) for unit cost.


LHC era

No Exponential is Forever … but We Can Delay 'Forever‘ftp://download.intel.com/ research/silicon/Gordon_Moore_ISSCC_021003.pdf

Technical Progress

Technology BoundariesTechnical Progress

Technology Boundaries

What are the limits on CPU?Moore’s Law

Currently CPU performance doubles every two years (or so) for unit cost.


Model 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Disk Doubling AGGRESSIVE 12 12 15 15 15 18 18 18 18 18 18

CPU Doubling STANDARD 24 24 24 24 24 24 24 24 24 24 24Table-1: Moore's Law Assumptions

Step-1

Calender Year

Select Hardware Price Assumptions using the GREEN cells (These are contained in the worksheet called "Assumptions")

AGGRESSIVE

STANDARD

Step-1..financial planning

Applies to our problem?(See Dave’s talk)

Reqts 2008 ALICE ATLAS CMS LHCb SUMCPU (kSI2K) 9100 16600 12600 9500 47800Disk (Tbytes) 3000 9200 8700 1300 22200Tape (Pbytes) 3.6 6 6.6 0.4 16.6Number of T1s 5 11 7 6 29Step-2.. Compare to

(e.g. Tier-1) expt. requirementsStep-3.. Conclude that morethan one centre is neededStep-4.. A Grid?

Ian Foster / Carl Kesselman: "A computational Grid is a

hardware and software infrastructure that provides dependable, consistent, pervasive and inexpensive access to high-end computational capabilities."

Currently network performance doubles every year (or so) for unit cost.


How do I start? http://www.gridpp.ac.uk

/start/

• Getting started as a Grid user• Quick start guide for LCG2

GridPP guide to starting as a user of the Large Hadron Collider Computing Grid.

• Getting an e-science certificateIn order to use the Grid you need a Grid certificate. This page introduces the UK e-Science Certification Authority, which issues cerficates to users. You can get a certificate from here.

• Using the LHC Computing Grid (LCG)CERN's guide on the steps you need to take in order to become a user of the LCG. This includes contact details for support.

• LCG user scenarioThis describes in a practical way the steps a user has to follow to send and run jobs on LCG and to retrieve and process the output successfully.

• Currently being improved.. DTEAM


Where do we start? Issues

https://edms.cern.ch/file/495809/2.2/LCG2-Limitations_and_Requirements.pdf

First large-scale Grid production problems being addressed…at all levels

“LCG-2 MIDDLEWARE

PROBLEMS ANDREQUIREMENTS

FOR LHC EXPERIMENT DATA

CHALLENGES”

Overall efficiency ~60%

¼ of the problems

¾ of the problems


GridPP Deployment Status (9-28-30/1/05)

Three Grids on Global scale in HEP (similar functionality)

sites CPUs• LCG (GridPP) 90 (16) 9000

(2242)• Grid3 [USA] 29 2800• NorduGrid 30 3200

GridPP deployment is part of LCG(Currently the largest Grid in the world)The future Grid in the UK is dependent upon LCG releases

totalCPU

freeCPU

runJob

waitJob

seAvail TB

seUsed TB

maxCPU

avgCPU

Total

2242 915 591 784 936.87 4.45 10648 2232


UK Tier-1/A Centre Rutherford Appleton

Laboratory

• High quality data services• National and international

role• UK focus for international

Grid development1000 CPU200 TB Disk60 TB Tape

(Capacity 1PB)

Grid Resource Discovery Time = 8 Hours

2004 CPU Utilisation2004 Disk Use

Peak Utilisation

Fall-off in Q4


UK Tier-2 Centres

The whole is better than the sum of the parts..


Level-2 Grid

In future will include services to facilitate collaborative (grid) computing•Authentication (PKI X509)•Job submission/batch service•Resource brokering•Authorisation•Virtual Organisation management •Certificate management•Information service•Data access/integration (SRB/OGSA-DAI/DQPS)•National Registry (of registry’s)•Data replication•Data caching•Grid monitoring•Accounting

* LeedsManchester *

* OxfordRAL *

* DL


Middleware Development

Configuration Management

Storage Interfaces

Network Monitoring

Security

Information Services

Grid Data Management

Deployment Area1. LCFG2. Generic3. Quattor

LHCC Comprehensive Review – November 2004 19

Enabling Grids for E-sciencE

INFSO-RI-508833

Prototype MiddlewareStatus & Plans (I)

• Workload Management– AliEn TaskQueue– EDG WMS (plus new TaskQueue and Information

Supermarket)– EDG L&B

• Computing Element– Globus Gatekeeper + LCAS/LCMAPS

Dynamic accounts (from Globus)

– CondorC– Interfaces to LSF/PBS (blahp)– “Pull components”

AliEn CE gLite CEmon (being configured)

Blue: deployed on development testbed

Red: proposed



INFSO-RI-508833

Prototype MiddlewareStatus & Plans (II)

• Storage Element– Existing SRM implementations

dCache, Castor, … FNAL & LCG DPM

– gLite-I/O (re-factored AliEn-I/O)• Catalogs

– AliEn FileCatalog – global catalog– gLite Replica Catalog – local

catalog– Catalog update (messaging)– FiReMan Interface– RLS (globus)

• Data Scheduling– File Transfer Service

(Stork+GridFTP)– File Placement Service – Data Scheduler

• Metadata Catalog– Simple interface defined

(AliEn+BioMed)

• Information & Monitoring– R-GMA web service version;

multi-VO support



INFSO-RI-508833

Prototype MiddlewareStatus & Plans (III)

• Security– VOMS as Attribute Authority

and VO mgmt– myProxy as proxy store– GSI security and VOMS

attributes as enforcement fine-grained authorization

(e.g. ACLs) globus to provide a set-uid

service on CE

• Accounting– EDG DGAS (not used yet)

• User Interface– AliEn shell– CLIs and APIs– GAS

Catalogs Integrate remaining services

• Package manager– Prototype based on AliEn

backend– evolve to final architecture

agreed with ARDA team


Middleware & OGSA-compliance

We need an “open” “grid” “services” “architecture”

1. Infrastructure Services that enable communication between disparate resources (computer, storage, applications, etc.), removing barriers associated with shared utilization.

2. Resource Management Services that enable the monitoring, reservation, deployment, and configuration of grid resources based on quality of service requirements

3. Data Services that enable the movement of data where it is needed – managing replicated copies, query execution and updates, and transforming data into new formats if required.

4. Context Services that describe the required resources and usage policies for each customer that utilizes the grid – enabling resource optimization based on service requirements.

5. Information Services that provide efficient production of, and access to, information about the grid and its resources, including status and availability of a particular resource.

6. Self-Management Services that support the attainment of stated levels of service with as much automation as possible, to reduce the costs and complexity of managing the system.

7. Security Services that enforce security policies within a virtual organization, promoting safe resource-sharing and appropriate authentication and authorization of users.

8. Execution Management Services that enable both simple and more complex workflow actions to be executed, including placement, provisioning, and management of the task lifecycle


Oasis WS-RF & WS-I+

• WS-RF (the Oasis standard) WS-I+ (implementation?)

• UK e-Science Core programme services (July 2004): WS-I+ – WS-I Basic Profile (XSD, WSDL 1.1, SOAP 1.1, UDDI)– WS-I Basic Security Profile (parts of WS-Security) BPELWS-Addressing (to be replaced the ongoing W3C activity).WS-ReliableMessagingWS-Eventing

• A service built with WS-RF will not interoperate with WS-I+ client… UK e-Science meeting today


gLite & ARDA Metadata

gLite (a standard?) ARDA (an implementation?)

End-user throughput or standards driven? GSOAP optimisation importantEarly days.. Some overlapping functionality – missing extensibility in gLite

APIs differ Testing ongoing: middle ground – adapt to gLite interfaces (e.g. AMI-

gLite), test ARDA implementation


The Oasis:OGSA:WSRF:WSI+:gLite:ARDA:ex

periment experiment:ARDA:gLite:WSI+:WSRF:OG

SA:Oasis

food chain?

1. A hierarchy? 2. A virtuous? circle

Only works if there is sufficient decomposition…Discussion required

Depends on yourWorld view…

Conference xxx - August 2003

Fabrizio GagliardiDataGrid Project Manager and EGEE designated Project Director

CERN

Geneva

Switzerland

Workshop on eInfrastructures (Internet and Grids)Best practices and challenges

Need to relate high level plan to what is required on the ground


LCG Robustness e.g. data

management• LCG File Catalog (LFC) developed to address the performance and scalability problems seen in the 2004 Data Challenges

• Features include hierarchical namespace, transactions, cursors, timeouts & retries, GSI security, ACLs...

• Performance testing almost complete• Tests of insert, query and delete rates

up to 40,000,000 entries and 10 clients / 100 concurrent threads

• Insert rates almost independent of number of entries in LFC, much more scalable than EDG RLS.

• Higher delete rate than EDG RLS• Query rate lower than Globus but

higher than EDG.. but LFC retrieves much more information with query so matches user patterns better

• Scales well to many replicas and LFNs per GUID, and to many concurrent users

-5000000 5000000 15000000 25000000 35000000 450000000

10

20

30

40

50

60

70

80

No. of entries in LFC

Mean

in

sert

tim

e (

ms)

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 500

50

100

150

200

250

300

Operation Rates, LFC with 1 M Entries, Single Client with Multiple Threads

Add rate

Delete rate

Query rate

Number of Threads

Ope

ratio

ns p

er s

econ

d

http://ppewww.ph.gla.ac.uk/~caitrian/LFC


Testing & Documentation

e.g. data management• lcg-aa "add-alias" adds an alias in RMC for

a given GUID. • lcg-cp "copy" copies a grid file to a specific

location on UI area. • lcg-cr "copy-and-register" copy a file to the

SE and registers the file in the SE's LRC. • lcg-del "delete" deletes a file. • lcg-gt "get-turl" gets the TURL for a given

SURL + transfer protocol. • lcg-infosites "list-all sites information" lists

important information for all sites on the grid.

• lcg-la "list-aliases" lists all the aliases for a given LFN, GUID or SURL.

• lcg-lg "list-GUID" lists the GUID for a given LFN or SURL.

• lcg-lr "list-replicas" lists the replicas for a given LFN, GUID or SURL.

• lcg-ra "remove-alias" removes an alias in RMC for a given GUID.

• lcg-rep "replicate" copies a file from one SE to another SE and registers it in the destination SE's LRC.

• lcg-rf "register-file" registers in LRC a file residing on an SE.

• lcg-uf "unregister-file" unregisters in LRC a file residing on an SE.

Preliminary testscompleted for all91 data managementCommandsSimple additional Documentation added

http://ppewww.ph.gla.ac.uk/~fergusjk/


Application Development

ATLAS LHCb CMS

BaBar (SLAC) SAMGrid (FermiLab)QCDGrid PhenoGrid


ApplicationsThere is a (slightly wonky?)

wheelUse it to get to where you need

to beZEUS uses LCG•needs the Grid to respond to increasing demand for MC production• up to 6 million Geant events per week on Grid since August 2004

1. The system developed for the large LHC experiments works (more) effectively for other (less resource-intensive) applications

2. Experiments need to work together with deployment team/sites

3. The de-facto deployment standard is LCG – it ~works. We can add components as required, to meet each experiment’s needs


Disseminationmuch has happened..

more people are reading about it..

GridPP2 gets its first term report Fri 28 Jan 2005 BaBar UK moves into the Grid era Tue 11 Jan 2005 LHCb-UK members get up to speed with the Grid Wed 5 Jan 2005

GridPP in Pittsburgh Thu 9 Dec 2004GridPP website busier than ever Mon 6 Dec 2004Optorsim 2.0 released Wed 24 Nov 2004ZEUS produces 5 million Grid events Mon 15 Nov 2004CERN 50th anniversary reception Tue 26 Oct 2004GridPP at CHEP'04 Mon 18 Oct 2004LHCb data challenge first phase a success for LCG and UK Mon 4 Oct 2004Networking in Nottingham - GLIF launch meeting Mon 4 Oct 2004GridPP going for Gold - website award at AHM Mon 6 Sep 2004GridPP at the All Hands Meeting Wed 1 Sep 2004R-GMA included in latest LCG release Wed 18 Aug 2004LCG2 administrators learn tips and tricks in Oxford Tue 27 Jul 2004Take me to your (project) leader Fri 2 Jul 2004ScotGrid's 2nd birthday: ScotGrid clocks up 1 million CPU hours Fri 25 Jun 2004Meet your production manager Fri 18 Jun 2004GridPP10 report and photographs Wed 9 Jun 2004CERN recognizes UK's outstanding contribution to Grid computing Wed 2 Jun 2004UK particle physics Grid takes shape Wed 19 May 2004A new monitoring map for GridPP Mon 10 May 2004Press reaction to EGEE launch Tue 4 May 2004GridPP at the EGEE launch conference Tue 27 Apr 2004LCG2 released Thu 8 Apr 2004University of Warwick joins GridPP Thu 8 Apr 2004Grid computing steps up a gear: the start of EGEE Thu 1 Apr 2004EDG gets glowing final review Mon 22 Mar 2004Grids and Web Services meeting, 23 April, London Tue 16 Mar 2004EU DataGrid Software License approved by OSI Fri 27 Feb 2004GridPP Middleware workshop, March 4-5 2004, UCL Fri 20 Feb 2004Version 1.0 of the Optorsim grid simulation tool released by EU DataGrid Tue 17 Feb 2004Summary and photographs of the 9th GridPP Collaboration Meeting Thu 12 Feb 2004

138,976 hitsin December


Annual data storage:2.4-2.8 PetaBytesper year? (~20%)

10 Million SPECint2000

10,000 PCs (3 GHz Pentium 4)

CD stack(~ 4 km)

The UK mountain climb has started..

Quantitatively, we’re ~10% of the way there in terms of

UK CPU (~1,000 ex ~10,000) and disk (~1 ex ~10 PB)

In production

terms, left base camp

We are here(0.4 km)

step-by-step plan in place…

For the Ben Nevis climb?

0. Production Grid

1. 1 2.1 3.1 4.1 5.1 6.1

1. 2 2.2 3.2 4.2 5.2 6.2

1. 3 2.3 3.3 4.3 6.3

1. 4 2.4 3.4 4.4 6.4

2.5 3.5 4.5

Navigate down External link Link to goals

2.6 3.6 4.6

Network

Management

& MonitoringInformation PhenoGrid

KnowledgeTransfer

32

Management

Grid Deployment Security CMS UKQCD

Engagement

Grid Technology Workload LHCb D0

Computing Fabric Data & Storage Ganga CDF Deployment

Grid Operations

1 6M/S/N LHC Apps

54

GridPP2 GoalTo develop and deploy a large scale production quality grid in the UK for the use of the Particle Physics community

Tier-A Tier-1 Tier-2 Deployment Middleware Support Experiment Support

Interoperability

ATLAS Dissemination

Management ExternalLCG

Planning

Applications Metadata

Non-LHC Apps

BaBar

SAMGrid

LHC Deployment Portal

totalCPU

freeCPU

runJob

waitJob

seAvail TB

seUsed TB

maxCPU

avgCPU

Total

2242 915 591 784 936.87 4.45 10648 2232


Summary GRIDPP-PMB-40-EXEC

• The Grid is a reality• A project was/is needed • Under control• LCG2 support: SC case presn. 3/2/05• 16 UK sites are on the Grid

– MoUs, planning, deployment, monitoring

– each underway as part of GridPP2• Developments estd., R-GMA deployed• gLite designed inc. web services• Interfaces developed, testing phase• Area transformed• Incorporation in HEP programme..

• Introduction• Project Management• Resources• LCG• Deployment

– Tier-1/A production+ Tier-2 resources

• M/S/N• EGEE• Applications • Dissemination • Beyond GridPP2


Top 10 Issues?

1. Issues are the ones that your oversight committee tells you are issues?

2. Issues are long-term (endemic) problems -they were around 3 years ago?

3. Issues are wider than this? The ones you thought might be problems at the start? (but they were called challenges)


PPARC Oversight Committee Issues

1. GridPP may be underestimating the difficulty of engaging with each of the experiment teams. 2. Document with a plan to support UK physics analysis community in 2007 is needed. 3. Tier-1 allocation policy - define usage policy. i.e. what is the absolute scale? Are we under/over-committing from PPARC perspective? 4. Need to update GridPP2 Risk Register.5. OC requires the LCG funding case to be put to them before going to Science Committee. (This has been done)6. Get-fit plan on Production Metrics. How do we move from 60% to >90% and how will this be monitored in the UK.7. Nail down the metrics - no sensible values yet established. Iterations are required.8. Clarify probable direction of GridPP in terms of middleware.

Tony Doyle - University of Glasgow

2002 Challenges2002 Challenges

Complete rollout of TB-1 and plan future upgradesComplete rollout of TB-1 and plan future upgrades Reconvened ATF to work closely with applicationsReconvened ATF to work closely with applications Make TB-2 a successMake TB-2 a success Deploy and exploit Tier-1/ADeploy and exploit Tier-1/A Applications to make good use of testbedsApplications to make good use of testbeds Solve interoperability issuesSolve interoperability issues

We are part of many larger We are part of many larger collaborations/structures/groupings - we need to collaborations/structures/groupings - we need to collaborate/discuss engage here, andcollaborate/discuss engage here, and

Focus on implementation in the UK… this will tell us Focus on implementation in the UK… this will tell us what works (and what doesn’t) at any given point.what works (and what doesn’t) at any given point.



Data Management, Security and

Sharing

1. Software process2. Software efficiency3. Deployment

planning 4. Link centres

5. Share data

6. Manage data7. Install software8. Analyse data9. Accounting

10. Policies


Top 10 Issues?

• Three methods to identify issues: • "If you cannot measure it, you cannot improve

it." • Need to quantify end-to-end throughput…

measurements are important… • Tackle the issues as they present themselves• In a timely way… LHC data is imminent…• Is there a GridPP top 10? • Answer?: No (probably)

tony doyle - university of glasgow 31 january 2005gridpp12 collaboration meeting gridpp status...

Documents

grid file

given guid

file residing

lcglg listguid

lcginfosites list

lcgrf registerfile registers

lcgcp copy

lcgcr copy