01-Oct-00 H. F. Hoffmann, CERN-DG/DI 3
CERN, evolution of resources
0
200
400
600
800
1000
1200
1400
52-54 1956 1958 1960 1962 1964 1966 1968 1970 1972 1974 1976 1978 1980 1982 1984 1986 1988 1990 1992 1994 1996 1998 2000
Years
MCHF, 2000 prices
Personnel
300 GeV
Exploitation
EnergieLHC
LEP 2
LEP 1
ppbar
ISR
2M&BEBC
PSSC
1400
1200
1000
800
600
400
200
0
Expenditure at 2000 prices
CERN paid (2750 staff, 800 fellows, paid associates)
Unpaid visiting scientists - Users
Contributions
0
100
200
300
400
500
600
700
800
900
20 25 30 35 40 45 50 55 60 65 70 75 80
Staff
Users
Age distribution>5500 visitors, 2750 staff,>1200/year turnover (3.99)Excellent education
CONTRIBUTIONS TO THE CERN PROGRAMMES, (MCHF, 2000 PRICES)
0
200
400
600
800
1000
1200
1400
0
1000
2000
3000
4000
5000
6000
7000
Year1956195819601962196419661968197019721974197619781980198219841986198819901992199419961998
CERN Paid Total
Users
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 4
aéro
port
GenèveAtlas
CMS
Alice
LHCb
PS
19542000
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 5
Evolution of CERN Particle Accelerators
• PS, 1959, ~1 GeV (cm)• ISR, 1971, ~9 GeV (cm)• SPS, 1976,~4.5 GeV (cm)
• SppbarS 1981 ~90 GeV (cm)
• LEP, 1989, 80-209(?) GeV (cm) • LHC, 2005, ~2'000 GeV (cm)
("cm" center of mass of partons)
s.c.cavity
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 6
"Facilities"
81 cm Bubble Chamber PS 1962-68Constructed in France
Big European Bubble Chamber PS and SPS 1974-1985External muon identifier
SFM facility (magnet, vacuum chamber,SFMDetector) at the ISR, external detectors as specific triggers by external laboratories with help of CERN
MWPC, 1968,SFMD: 300K wires 1974DAQ, Trigger,large datarates
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 7
Experimental Apparatus
BEBC photo, v-beam, ~30 tracks, semi-automatic scanning,very sophisticated tracking and analysis codes---->computer literacy>computer literacyresolution ~ 100 µ, < 1 event/sec, 20m3 liquid superheated hydrogen, 1.5 Tesla S.C. magnet
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 8
"Counter" Experiments
M&G Fidecaro, 1964, spark chambers, -->
N* experiment at ISR, 1972
UA1 at SppbarS 1981-1989 ALEPH at LEP 1989-2000
Charm Search at ISR, 1975
ATLAS at LHC, 2005-2020150*106 sensors;
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 9
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 10
Experimental Apparatus, continued
Pixel detector50*100µ/pixel, 140million
channels
LHC collisions, 109 events/s Complexity of data: ~250SPECint95*sec/event 1 / 1013 selectivity
Basic building block in full pixel readout chip : 8 pixels/12 000 transistors in 400 by 425 mm2
General Purpose detector,Multiple, simultaneous detection modes, OO programming,millions of lines of code
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 11
ATLAS Collaboration
0
50
100
150
200
250
300
Austria
Czech Republic
DenmarkFinlandFranceGermany
Greece
Italy
Netherlands
NorwayPolandPortugal
Slovac Republic
SpainSweden
Switzerland
United Kingdom
CERNArmeniaAustralia
Azerbaijan RepublicRepublic of Belarus
BrazilCanada
China PR
Republic of Georgia
IsraelJapanMoroccoRomania
Russia
JINR Dubna
SloveniaTaiwanTurkey
United States
1800 physicists,150 institutes; 35 countriesR&D, proposal, design, reviews, approval: 1988-1996Construction, installation 1996-2005, operation 2005-2020Material Cost 300 M Euro, CERN part:20%
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 12
CERN's Network in the World
267 institutes in Europe, 4603 users208 institutes elsewhere, 1632 userssome points = several institutes
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 13
Virtual Room Videoconferencing System• 3267 machines
reg.• 1994 people in• 52 countries• 182 institutes• http://vrvs.cern.ch/
Bandwidth >256 Kbps--> >10 frames/sec
9
Virtual Rooms Concept
u Enter a Virtual Room Through Your NearestReflector
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 14
– Open, global collaboration of critical mass, able to deal with all problems posed, together with CERN and collaborating institutes
– "Lean, bottom-up" self-organisationself-organisation; success based on experienced collaborators, eager young people, common goals and competition
– MoU, best intentions but not legally bindingMoU, best intentions but not legally binding– Free choice of collaborating institutes to participate -
or not– Clear common long-term mission, clear objectives,– Free exchange of ideas, technologies, R&D results – Often best people in the field of interest– External peer reviews; elaborate internal reviews and
QA– Good record of achievements in terms of delivery to
specs, schedules, budgets
Some typical features of such collaborations
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 15
Basic Organisation of any Physics- , Scientific- Experiment
• Organigram– Hierarchy - Heterarchy
. . . .
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 16
ATLAS ORGANIZATION
(Chair : J.D. DowellDeputy : M. Cavalli-Sforza)
Plenary Meeting
Spokesperson(P. Jenni)
CB ChairAdvisory Group
Technical Co-ordinator
(M. Nessi)
Resource Co-ordinator
(P. Schmid)
Inner Detector(M. A. Parker
M. Tyndel)L. Rossi
LAr Calorimeter(D. FournierD. LissauerH. Oberlack)
Tile Calorimeter(M. Nessi)
(G. CiapettiC. Fabjan)
MuonSpectrometer
Electronics(H. Williams)
(N. Ellis)
Trigger
DAQ(L. Mapelli)
Software,Computing(J. Knobloch)
Physics andDetectorSim.
(D. Froidevaux)
Collaboration Board
Magnet(H.TenKate)
Executive Board
Resource ReviewBoard
(Deputy: T. Akesson)
Gen. Members(P. Le DuA. Zaitsev)
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 17
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 18
CMS COLLABORATION RRB CMS-D 98-31
Memorandum of Understandingfor Collaboration in the Construction of the CMS Detector
betweenThe EUROPEAN ORGANISATION FOR NUCLEAR RESEARCH,
andan Institution/Funding Agency of the CMS Collaboration
Preamble(a) A group of Institutes from CERN Member and non-Member States, and CERN,has agreed to collaborate to form the CMS Collaboration (Annex 1). This Collaboration has proposed to CERN an experiment to study particle interactions at the highest possible energies and luminosities to be reached with the Large Hadron Collider (LHC). These Institutes have secured the support of their Funding Agencies to enable them to participate in the CMS Collaboration.(b) Agreement to this Collaboration is effected through identical Memoranda of Understanding (hereafter referred to as MoU) between each Funding Agency or Institute, as appropriate, in the Collaboration and CERN, as the Host Laboratory. These MoUs define the Collaboration and its objectives, and the rights and obligations of the collaborating Institutes.(c) On the basis of a Technical Proposal submitted in December 1994 (CERN/ LHCC/94-38) and a detailed review of the scientific merits, the technological feasibility and estimates of the needed resources, the LHC Committee (LHCC) recommended approval of the experiment to the CERN Research Board, subject to a set of milestones to be met by the experiment in its initial phase (CERN/LHCC 95-76).(d) Based on the recommendation by the LHCC and in agreement with the list of milestones, the Research Board recommended to the Director General of CERN to approve the project, together with plans, including milestones, leading to the sub-detector Technical Design Reports.
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 19
(e) The Director General accepted the Research Board recommendation and approved the project to build the detector for the CMS experiment within a cost ceiling not exceeding 475 MCHF (in 1995 prices).
(f) Before proceeding to the final construction phase, each sub-detector (c.f.. Article 4.1) will be subjected to a technical, financial, and manpower review (CERN/DG/RB 95-234) by the LHCC based on the Technical Design Reports. This process will be completed during 1997 and 1998 for most of the sub-systems.
(g) A Resources Review Board (RRB) has been constituted which comprises the representatives of all CMS Funding Agencies and the managements of CERN and the CMS Collaboration. It is chaired by the CERN Director of Research.The role of the RRB includes :· reaching agreement on the Memorandum of Understanding· monitoring the Common Projects and the use of the Common Funds· monitoring the general financial and manpower support· reaching agreement on a maintenance and operation procedure andmonitoring its functioning· endorsing the annual construction and maintenance and operation budgets of the detector.The management of the Collaboration reports regularly to the RRB on technical, managerial, financial and administrative matters, and on the composition of the Collaboration.
(h) Interim MoUs become obsolete
(i) This MoU is not legally binding, but the Institutes and Funding Agencies recognize that the success of the Collaboration depends on all its members adhering to its provisions. Any default will be dealt with, in the first instance,by the Collaboration and if necessary then by the RRB.
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 20
ATLAS Organisation (as example)• Principles:
– Democracy– Separation of policy-making and executive powers– Minimal formal organisation– Limited terms of office
• Plenary Meeting:– Forum of the all-hands discussions, – all major decisions concerning physics objectives and results,
hardware and software design, organisational matters must be discussed there
• Collaboration Board:– Policy- and decision-making body with typical tasks:
• Decisions on global detector design• Policy matters wrt official bodies• Financial and human resources• Elections• Organisation and membership
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 21
Project Organisation (example ATLAS)
• ATLAS project=organised sum of sub-projects and parts, conceived, designed and fabricated to a variety of habits, standards, cultures around the world
• Engineering organisation -- "top cultural layer", common language and common "rules of the game" to permit "engineering communication" throughout the project– Project breakdown structure (product, assembly breakdowns)– Work packages and WP-descriptions– Schedules, milestones, reporting for project follow up– Quality assurance
• Reviews at the various project stages like design, construction, assembly
• Configuration management
– Integration, mechanical, services, "environmental", accelerator
– Safety (CERN-LHCC/99-01; ATLAS TDR 13, "Tech. Co-ordination"; 31-01-1999, 598
pages)
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 22
From Particles to Petabytes:Challenges in High Throughput Computing
Example:(Data-) Grid,(EU-Project, NSF Project,..)
a global Particle Physics Projectto make world-wide LHC computing
possible
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 23
On-line System• Multi-level trigger• Filter out less interesting• Reduce data volume• 24 x 7 operation
Level 1 - Special Hardware
Level 2 - Embedded Processors
40 MHz40 MHz (1000 TB/sec) equivalent)
(1000 TB/sec) equivalent)
Level 3 – Farm of commodity CPU
75 KHz75 KHz (75 GB/sec)fully digitised
(75 GB/sec)fully digitised5 KHz5 KHz (5 GB/sec)
(5 GB/sec)100 Hz100 Hz (100 MB/sec)
(100 MB/sec)
Data Recording &
Data Recording &
Offline Analysis
Offline Analysis
Digital telephone1-2 KB/sec
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 24
1 billion people 1 billion people surfing the Websurfing the Web
1 billion people 1 billion people surfing the Websurfing the Web
How Much Data is Involved?
105
104104
103103
102102
Level 1 Rate (Hz)
Level 1 Rate (Hz)
High Level-1 Trigger(1 MHz)High Level-1 Trigger(1 MHz)
High No. ChannelsHigh Bandwidth(500 Gbit/s)
High No. ChannelsHigh Bandwidth(500 Gbit/s)
High Data Archive(PetaByte)High Data Archive(PetaByte)
LHCBLHCB
KLOE
HERA-BHERA-B
CDF IICDF II
CDFCDF
H1ZEUS
H1ZEUS
UA1UA1
LEPLEP
NA49NA49
ALICEALICE
Event Size (bytes)Event Size (bytes)
104104 105105 106106
ATLASCMSATLASCMS
106106
107107
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 26
Complex Queries = More CPU Per Byte
Evolution of Computing CapacityEvolution of Computing Capacity
Thousands ofThousands ofSPECint 95SPECint 95
00100100200200300300400400500500600600700700800800900900
1'0001'000
1997
1997
1998
1998
1999
1999
2000
2000
2001
2001
2002
2002
2003
2003
2004
2004
2005
2005
YearYear
OthersOthers
COMPASSCOMPASS
LHCLHCCERN 1999:CERN 1999:
3.5K SI953.5K SI95900 CPUs900 CPUs
CERN 1999:CERN 1999:3.5K SI953.5K SI95900 CPUs900 CPUs
?
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 28
CERN Computer Center Today…--> Commodity
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 29
• No longer aligned with supercomputing philosophies
CERN Computer Center Today…
Therefore, our natural affinity has shifted from Therefore, our natural affinity has shifted from supercomputerssupercomputers
towards ISPs, e-commerce and data marketerstowards ISPs, e-commerce and data marketers
• Require many small independent problem solutions
– “High Throughput Computing” processing “click-like” interactions in parallel
• A marriage of supercomputer storage systems with supermarket commodity CPU
– Disk access layer (hw+sw) sandwiched between
– “Middle-ware” on network layer important
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 31
On-demand creation of powerfulOn-demand creation of powerfulvirtual computing and data systemsvirtual computing and data systems
Grid: Flexible, high-performance access to all significant resources
Sensor nets
http://
http://
Web: Uniform access to HTML documents
Data Stores
Computers
Softwarecatalogs
Colleagues
Grids: Next Generation Web
Web-sites
0
50
100
150
200
250
300
Austria
Czech Republic
DenmarkFinlandFranceGermany
Greece
Italy
Netherlands
NorwayPolandPortugal
Slovac Republic
SpainSweden
Switzerland
United Kingdom
CERNArmeniaAustralia
Azerbaijan RepublicRepublic of Belarus
BrazilCanada
China PR
Republic of Georgia
IsraelJapanMoroccoRomania
Russia
JINR Dubna
SloveniaTaiwanTurkey
United States
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 32
LHC Vision: Data Grid Hierarchy
Tier 1
Tier2 Center
Online System
Offline Farm,CERN Computer
Ctr > 20 TIPS
German Centre
FNAL Center Italy Center UK Center
InstituteInstituteInstituteInstitute ~0.25TIPS
Workstations
~100 MBytes/sec
~2.5 Gbits/sec
100 - 1000
Mbits/sec
Bunch crossing per 25 nsecs; 100 triggers per second. Event is ~1 MByte in size
Physicists work on analysis “channels”
Each institute has ~10 physicists working on one or more channels
Physics data cache
~PByte/sec
~0.6-2.5 Gbits/sec
Tier2 CenterTier2 CenterTier2 Center
~622 Mbits/sec
Tier 0 +1
Tier 3
Tier 4
Tier2 Center Tier 2
Experiment
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 33
The Grid Middleware Services Concept• Standard services that
– Provide uniform, high-level access to a wide range of resources (including networks)
– Address interdomain issues: security, policy
– Permit application-level management and monitoring of end-to-end performance
• Broadly deployed, like Internet Protocols
• Enabler of application-specific tools as well as applications themselves
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 37
GEANT, necessary infrastructure
Minimum bandwidth of 2.5 Gbps between core nodes, possibility of starting with some 10Gbps (STM-64/OC-768c) circuits is not excluded.
Connection to other World Regions in principle via core nodes only, They will, together, form a European Distributed Access (EDA) “point” conceptually similar to the STAR TAP.
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 39
The Web, a historical case study• Invented at CERN in 1989 as application layer
on top of the internet infrastructure• Development started in Europe (small) and US
(big, >50 computer scientists initially for MOSAIC)
• 80% of the most visited sites: US, <10% Europe
Web Site Servers
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 42
Questions and AnswersQ1:Flexibility of institutional structure to allow researchers to change field
Q2: Exchange of knowledge across disciplines and institutions
Q3: Main obstacles to international co-operation
Q4:Information revolution
A1:Beginning discussion with astro-, space- physics; technology transfer-> funding for interdisciplinary activities? However,very clear mission, "mono-culture"A2: More fellowships in technological and interdisciplinary fields with specific funding: domain competence and add-on competence(KI, DataGrid)A3: Funding agencies, scientists, politicians still think "national" Employment conditions, spouses, schools--> keep national employment in Europe "in exchange" plus adjustment allowance, help for spouses to find appropriate work, international schoolsA4: Promote e-science, grids, . . .
01-Oct-00 H. F. Hoffmann, CERN-DG/DI 43
International Collaboration Objective: top scientific excellence in your particular field
Add resources("Resources": Talent; particular knowledge, experience, methods; specific scientific apparatus, technologies; funds)
Create a complete, competitive, technological infrastructure in your particular field beyond local, regional, national meansCreate a network of competence to solve detailed problems quickly and to prepare new meansBase your collaboration to a large degree on Universities (talent)
Reach, sustain excellence by attracting the best people
"Nobody is perfect" - but a team can be