henp data grids and startap henp data grids and startap worldwide analysis at regional centers...
TRANSCRIPT
HENP DATA GRIDS HENP DATA GRIDS and STARTAPand STARTAP
Worldwide Analysis at Regional CentersWorldwide Analysis at Regional Centers Harvey B. Newman (Caltech)Harvey B. Newman (Caltech)
HPIIS ReviewHPIIS ReviewSan Diego, October 25, 2000San Diego, October 25, 2000
http://l3www.cern.ch/~newman/hpiis2000.ppthttp://l3www.cern.ch/~newman/hpiis2000.ppt
Next Generation Experiments: Next Generation Experiments: Physics and Technical GoalsPhysics and Technical Goals
The extraction of small or subtle new “discovery” The extraction of small or subtle new “discovery” signals from large and potentially overwhelming signals from large and potentially overwhelming backgrounds; or “precision” analysis of large samplesbackgrounds; or “precision” analysis of large samples
Providing rapid access to event samples and subsets Providing rapid access to event samples and subsets from massive data stores, from ~300 Terabytes in 2001 from massive data stores, from ~300 Terabytes in 2001 Petabytes by ~2003, ~10 Petabytes by 2006, to ~100 Petabytes by ~2003, ~10 Petabytes by 2006, to ~100 Petabytes by ~2010.Petabytes by ~2010.
Providing analyzed results with rapid turnaround, byProviding analyzed results with rapid turnaround, bycoordinating and managing the coordinating and managing the LIMITED LIMITED computing, computing, data handling and network resources effectivelydata handling and network resources effectively
Enabling rapid access to the data and the collaboration, Enabling rapid access to the data and the collaboration, across an ensemble of networks of varying capability, across an ensemble of networks of varying capability, using heterogeneous resources.using heterogeneous resources.
The Large Hadron Collider (2005-)The Large Hadron Collider (2005-)
A next-generation particle collider A next-generation particle collider
the largest superconductor installation in the largest superconductor installation in the worldthe world
A bunch-bunch collision will take place A bunch-bunch collision will take place every 25 nanoseconds: each generating ~20 every 25 nanoseconds: each generating ~20 interactionsinteractions
But only one in a trillion may lead to a But only one in a trillion may lead to a major physics discovery major physics discovery
Real-time data filtering: Real-time data filtering: Petabytes per second to Gigabytes per Petabytes per second to Gigabytes per secondsecond
Accumulated data of many Petabytes/YearAccumulated data of many Petabytes/Year
Large data samples explored and analyzed Large data samples explored and analyzed by thousands of geographically dispersed by thousands of geographically dispersed scientists, in hundreds of teamsscientists, in hundreds of teams
Computing Challenges: Computing Challenges: LHC ExampleLHC Example
Geographical dispersion:Geographical dispersion: of people and resources of people and resources Complexity:Complexity: the detector and the LHC environment the detector and the LHC environment Scale: Scale: Tens of Petabytes per year of dataTens of Petabytes per year of data
1800 Physicists 150 Institutes 34 Countries
Major challenges associated with:Major challenges associated with:Communication and collaboration at a distanceCommunication and collaboration at a distance
Network-distributed computing and data resources Network-distributed computing and data resources Remote software development and physics analysisRemote software development and physics analysisR&D: New Forms of Distributed Systems: Data GridsR&D: New Forms of Distributed Systems: Data Grids
Four LHC Experiments: The Four LHC Experiments: The Petabyte to Exabyte Petabyte to Exabyte
ChallengeChallengeATLAS, CMS, ALICE, LHCBATLAS, CMS, ALICE, LHCB
Higgs + New particles; Quark-Gluon Plasma; CP ViolationHiggs + New particles; Quark-Gluon Plasma; CP Violation
Data written to tapeData written to tape ~25 Petabytes/Year and ~25 Petabytes/Year and UP;UP; 0.25 Petaflops and UP 0.25 Petaflops and UP
0.1 to 1 Exabyte (1 EB = 100.1 to 1 Exabyte (1 EB = 101818 Bytes) Bytes) (~2010) (~2015 ?) Total for the LHC Experiments(~2010) (~2015 ?) Total for the LHC Experiments
Higgs SearchLEPC September 2000
All charged tracks with pt > 2 GeV
Reconstructed tracks with pt > 25 GeV
(+30 minimum bias events)
109 events/sec, selectivity: 1 in 1013 (1 person in a thousand world populations)
LHC: Higgs Decay into 4 muons LHC: Higgs Decay into 4 muons (tracker only); 1000X LEP Data Rate(tracker only); 1000X LEP Data Rate
On-line Filter SystemOn-line Filter System Large variety of triggers Large variety of triggers
and thresholds: select and thresholds: select physics à la cartephysics à la carte
Multi-level triggerMulti-level trigger Filter out less Filter out less
interestinginterestingeventsevents
Online reduction 10Online reduction 1077
Keep highly selected Keep highly selected eventsevents
Result: PetabytesResult: Petabytesof Binary Compactof Binary CompactData Per YearData Per Year
Level 1Level 1 - Special Hardware
- Special Hardware
Level 2Level 2 - Processors
- Processors
40 MHz
40 MHz (1000 TB/sec) equivalent)
(1000 TB/sec) equivalent)
Level 3Level 3 – Farm of Commodity CPUs
– Farm of Commodity CPUs
75 KHz 75 KHz (75 GB/sec)fully digitised
(75 GB/sec)fully digitised5 KHz5 KHz (5 (5 GB/sec)GB/sec)100 Hz
100 Hz (100 MB/sec)
(100 MB/sec)
Data Recording &
Data Recording &
Offline Analysis
Offline Analysis
LHC Vision: Data Grid HierarchyLHC Vision: Data Grid Hierarchy
Tier 1
Tier2 Center
Online System
Offline Farm,CERN Computer
Ctr > 20 TIPS
FranceCentre
FNAL Center Italy Center UK Center
InstituteInstituteInstituteInstitute ~0.25TIPS
Workstations
~100 MBytes/sec
~2.5 Gbits/sec
100 - 1000
Mbits/sec
Physicists work on analysis “channels”
Each institute has ~10 physicists working on one or more channels
Physics data cache
~PByte/sec
~0.6-2.5 Gbits/sec
Tier2 CenterTier2 CenterTier2 Center
~622 Mbits/sec
Tier 0 +1
Tier 3
Tier 4
Tier2 Center Tier 2
Experiment
Why Worldwide Computing? Regional Why Worldwide Computing? Regional Center Concept AdvantagesCenter Concept Advantages
Managed, fair-shared access for Physicists everywhereManaged, fair-shared access for Physicists everywhere Maximize total funding resources while meeting the Maximize total funding resources while meeting the
total computing and data handling needstotal computing and data handling needs Balance between proximity of datasets to appropriate Balance between proximity of datasets to appropriate
resources, and to the usersresources, and to the users Tier-N ModelTier-N Model
Efficient use of network: higher throughputEfficient use of network: higher throughput Per Flow: Local > regional > national > internationalPer Flow: Local > regional > national > international
Utilizing all intellectual resources, in several time zonesUtilizing all intellectual resources, in several time zones CERN, national labs, universities, remote sitesCERN, national labs, universities, remote sites Involving physicists and students at their home institutionsInvolving physicists and students at their home institutions
Greater flexibility to pursue different physics interests, Greater flexibility to pursue different physics interests, priorities, and resource allocation strategies by regionpriorities, and resource allocation strategies by region
And/or by Common Interests (physics topics, subdetectors,…)And/or by Common Interests (physics topics, subdetectors,…) Manage the System’s ComplexityManage the System’s Complexity
Partitioning facility tasks, to manage and focus resourcesPartitioning facility tasks, to manage and focus resources
Grid Services Architecture [*]Grid Services Architecture [*]
GridGridFabricFabric
GridGridServicesServices
ApplnApplnToolkitsToolkits
ApplnsApplns
Data stores, networks, computers, display Data stores, networks, computers, display devices,… ; associated local servicesdevices,… ; associated local services
Protocols, authentication, policy, resource Protocols, authentication, policy, resource management, instrumentation, discovery,etc.management, instrumentation, discovery,etc.
......RemotRemot
eevizviz
toolkittoolkit
RemotRemotee
comp.comp.toolkittoolkit
RemotRemotee
datadatatoolkittoolkit
RemotRemotee
sensorssensorstoolkittoolkit
RemotRemotee
collab.collab.toolkittoolkit
A Rich Set of HEP Data-Analysis A Rich Set of HEP Data-Analysis Related ApplicationsRelated Applications
[*] [*] Adapted from Ian FosterAdapted from Ian Foster
SDSS Data Grid (In GriPhyN): SDSS Data Grid (In GriPhyN): A Shared VisionA Shared Vision
Three main functions:Three main functions: Raw data processing on a Grid (FNAL)Raw data processing on a Grid (FNAL) Rapid turnaround with TBs of dataRapid turnaround with TBs of data Accessible storage of all image dataAccessible storage of all image data
Fast science analysis environmentFast science analysis environment(JHU)(JHU)
Combined data access + analysis Combined data access + analysis of calibrated dataof calibrated data
Distributed I/O layer and processing Distributed I/O layer and processing layer; shared by whole collaborationlayer; shared by whole collaboration
Public data accessPublic data access SDSS data browsing for SDSS data browsing for
astronomers, and studentsastronomers, and students Complex query engine for the publicComplex query engine for the public
RD45, RD45, GIODGIOD Networked Object DatabasesNetworked Object Databases Clipper/GC Clipper/GC High speed access to Objects or File data High speed access to Objects or File data FNAL/SAM FNAL/SAM for processing and analysisfor processing and analysis SLAC/OOFS SLAC/OOFS Distributed File System + Objectivity Distributed File System + Objectivity Interface Interface NILE, Condor:NILE, Condor: Fault Tolerant Distributed ComputingFault Tolerant Distributed Computing
MONARCMONARC LHC Computing Models: LHC Computing Models: Architecture, Simulation, Strategy, PoliticsArchitecture, Simulation, Strategy, Politics
ALDAPALDAP OO Database Structures & Access Methods OO Database Structures & Access Methods for Astrophysics and HENP Datafor Astrophysics and HENP Data
PPDGPPDG First Distributed Data Services and First Distributed Data Services and Data Grid System PrototypeData Grid System Prototype
GriPhyN GriPhyN Production-Scale Data GridsProduction-Scale Data Grids EU Data GridEU Data Grid
Roles of ProjectsRoles of Projectsfor HENP Distributed Analysisfor HENP Distributed Analysis
GIOD: Globally InterconnectedGIOD: Globally InterconnectedObject DatabasesObject Databases
Hit
Track
Detector
MultiTB OO Database MultiTB OO Database Federation; used across Federation; used across LANs and WANsLANs and WANs
170 MByte/sec CMS 170 MByte/sec CMS MilestoneMilestone
Developed Java 3D OO Developed Java 3D OO Reconstruction, Analysis Reconstruction, Analysis and Visualization and Visualization Prototypes that Work Prototypes that Work Seamlessly OverSeamlessly OverWorldwide NetworksWorldwide Networks
Deployed facilities and Deployed facilities and database federations as database federations as testbedstestbeds for Computing for Computing Model studiesModel studies
The Particle Physics Data Grid The Particle Physics Data Grid (PPDG)(PPDG)
First Round Goal: First Round Goal: Optimized cached read access to 10-100 Gbytes Optimized cached read access to 10-100 Gbytes drawn from a total data set of 0.1 to ~1 Petabytedrawn from a total data set of 0.1 to ~1 Petabyte
PRIMARY SITEPRIMARY SITEData Acquisition,Data Acquisition,
CPU, Disk, CPU, Disk, Tape RobotTape Robot
SECONDARY SITESECONDARY SITECPU, Disk, CPU, Disk, Tape RobotTape Robot
Site to Site Data Replication Service
100 Mbytes/sec
ANL, BNL, Caltech, FNAL, JLAB, LBNL, ANL, BNL, Caltech, FNAL, JLAB, LBNL, SDSC, SLAC, U.Wisc/CSSDSC, SLAC, U.Wisc/CS
Multi-Site Cached File Access Service
UniversityUniversityCPU, Disk, CPU, Disk,
UsersUsers
PRIMARY SITEPRIMARY SITEDAQ, Tape, DAQ, Tape,
CPU, CPU, Disk, RobotDisk, Robot
Satellite SiteSatellite SiteTape, CPU, Tape, CPU, Disk, RobotDisk, Robot
UniversityUniversityCPU, Disk, CPU, Disk,
UsersUsers
UniversityUniversityCPU, Disk, CPU, Disk,
UsersUsers
UniversityUniversityCPU, Disk, CPU, Disk,
UsersUsers
UniversityUniversityCPU, Disk, CPU, Disk,
UsersUsers
Satellite SiteSatellite SiteTape, CPU, Tape, CPU, Disk, RobotDisk, Robot
Matchmaking, Co-Scheduling: SRB, Condor, Globus services; HRM, NWSMatchmaking, Co-Scheduling: SRB, Condor, Globus services; HRM, NWS
PPDG WG1: Request ManagerPPDG WG1: Request Manager
tape system
HRM
Replicacatalog
NetworkWeatherServicePhysical file
transfer requests GRID
RequestInterpreter
DiskCache
Event-file Index
DRM
DiskCache
RequestExecutor
Logical Set of Files Request
Planner(Matchmaking)DRMDisk
Cache
CLIENT CLIENT
Logical Request
REQUEST MANAGER
LLNL
Earth Grid System Prototype Earth Grid System Prototype Inter-communication DiagramInter-communication Diagram
Disk
Client
Request Manager
ISIGSI-
wuftpd
Disk
SDSCGSI-pftpd
HPSS
LBNLGSI-wuftpd
Disk
ANLGSI-
wuftpd
Disk
NCARGSI-
wuftpd
Disk
LBNL
Diskon
Clipper
HPSS
HRM
ANLReplica Catalog
GIS with NWS
GSI-ncftp
GS
I-ncftpGSI-n
cftp
LDAP Script
LDAP C API or Script
GSI-ncftp
GSI-ncftpGSI-ncftp CORBA
Grid Data Management Grid Data Management Prototype (GDMP)Prototype (GDMP)
Distributed Distributed Job ExecutionJob Execution and and
Data Handling:Data Handling: TransparencyTransparency PerformancePerformance Security Security Fault ToleranceFault Tolerance AutomationAutomation
Submit job
Replicate data
Replicatedata
Site A Site B
Site C
Jobs are executed locally or
remotely Data is always
written locally Data is replicated
to remote sites
Job writes data locally
GDMP V1.1: Caltech + EU DataGrid WP2 Tests by CALTECH, CERN, FNAL, Pisa for CMS “HLT” Production 10/2000;
Integration with ENSTORE, HPSS, Castor
GriPhyN: Grid Physics NetworkGriPhyN: Grid Physics Network
A New Form of Integrated Distributed SystemA New Form of Integrated Distributed System
Meeting the Scientific Goals Meeting the Scientific Goals of LIGO, SDSS and the LHC Experiments of LIGO, SDSS and the LHC Experiments
Focus on Tier2 Centers at UniversitiesFocus on Tier2 Centers at Universities In a Unified Hierarchical Grid of Five LevelsIn a Unified Hierarchical Grid of Five Levels
18 Centers; with Four Sub-Implementations 18 Centers; with Four Sub-Implementations 5 Each in US for LIGO, CMS, ATLAS; 3 for SDSS5 Each in US for LIGO, CMS, ATLAS; 3 for SDSS Near Term Focus on LIGO, SDSS handling of real data; Near Term Focus on LIGO, SDSS handling of real data;
LHC “Data Challenges” with simulated dataLHC “Data Challenges” with simulated data Cooperation with PPDG, MONARC and EU DataGridCooperation with PPDG, MONARC and EU DataGrid
http://www.phys.ufl.edu/~avery/GriPhyN/http://www.phys.ufl.edu/~avery/GriPhyN/
Data Intensive Data Intensive ScienceScience
GriPhyN: PetaScale Virtual Data GridsGriPhyN: PetaScale Virtual Data Grids
Virtual Data Tools
Request Planning &
Scheduling ToolsRequest Execution & Management Tools
Transforms
Distributed resources(code, storage,
computers, and network)
Resource Management
Services
Resource Management
Services
Security and Policy
Services
Security and Policy
Services
Other Grid ServicesOther Grid
Services
Interactive User Tools
Production TeamIndividual InvestigatorWorkgroups
Raw data source
EU DataGridEU DataGridhttp://www.cern.ch/gridhttp://www.cern.ch/grid
Organized by CERNOrganized by CERN HEP Participants: Czech Republic, France, Germany, HEP Participants: Czech Republic, France, Germany,
Hungary, Italy, Netherlands, Portugal, UK; (US)Hungary, Italy, Netherlands, Portugal, UK; (US) Industrial participationIndustrial participation Grid forum contextGrid forum context 12 Work Packages (One coordinator each)12 Work Packages (One coordinator each)
Middleware: Work scheduling; data management; Middleware: Work scheduling; data management; application monitoring; fabric management; application monitoring; fabric management; storage managementstorage management
Infrastructure: Testbeds and demonstrators; Infrastructure: Testbeds and demonstrators; advanced network servicesadvanced network services
Applications: HEP, Earth Observation; Biology Applications: HEP, Earth Observation; Biology
[*] [*] Basic Middleware Framework: GlobusBasic Middleware Framework: Globus
WorkPackageNumber
Work Package title Leadcontractor
WP1 Grid Workload Management INFN
WP2 Grid Data Management CERN
WP3 Grid Monitoring Services PPARC
WP4 Fabric Management CERN
WP5 Mass Storage Management PPARC
WP6 Integration Testbed CNRS
WP7 Network Services CNRS
WP8 High Energy Physics Applications CERN
WP9 Earth Observation Science Applications ESA
WP10 Biology Science Applications INFN
WP11 Dissemination and Exploitation INFN
WP12 Project Management CERN
EU DataGrid ProjectEU DataGrid ProjectWork PackagesWork Packages
Emerging Emerging Data GridData Grid User Communities User Communities
NSF Network for Earthquake Engineering NSF Network for Earthquake Engineering Simulation (NEES)Simulation (NEES) Integrated instrumentation, collaboration, Integrated instrumentation, collaboration,
simulationsimulation Grid Physics Network (GriPhyN)Grid Physics Network (GriPhyN)
ATLAS, CMS, LIGO, SDSSATLAS, CMS, LIGO, SDSS World-wide distributed analysis World-wide distributed analysis
of Petascale dataof Petascale data Access Grid; VRVS: supporting Access Grid; VRVS: supporting
group-based collaborationgroup-based collaborationAndAnd
Genomics, Proteomics, ...Genomics, Proteomics, ... The Earth System Grid and EOSDISThe Earth System Grid and EOSDIS Federating Brain DataFederating Brain Data Computed MicroTomography Computed MicroTomography …… NVO, GVONVO, GVO
GRIDs In 2000: SummaryGRIDs In 2000: Summary
Grids are changing the way we do science Grids are changing the way we do science and engineeringand engineering
From Computation to DataFrom Computation to Data Key services and concepts have been Key services and concepts have been
identified, and development has startedidentified, and development has started Major IT challenges remainMajor IT challenges remain
AnAn Opportunity & Obligation for HEP/CSOpportunity & Obligation for HEP/CSCollaborationCollaboration
Transition of services and applications to production Transition of services and applications to production use is starting to occuruse is starting to occur
In future more sophisticated integrated services and In future more sophisticated integrated services and toolsets (Inter- and IntraGrids+) could drive advances in toolsets (Inter- and IntraGrids+) could drive advances in many fields of science & engineeringmany fields of science & engineering
HENP, facing the need for Petascale Virtual Data, HENP, facing the need for Petascale Virtual Data, is both an early adopter, and a leading developer is both an early adopter, and a leading developer of Data Grid technologyof Data Grid technology
Bandwidth Requirements Bandwidth Requirements Projection (Mbps): ICFA-NTFProjection (Mbps): ICFA-NTF
1998 2000 2005
BW Utilized Per Physicist(and Peak BW Used)
0.05 - 0.25(0.5 - 2)
0.2 – 2(2-10)
0.8 – 10(10 – 100)
BW Utilized by a UniversityGroup
0.25 - 10 1.5 - 45 34 - 622
BW to a Home Laboratory OrRegional Center
1.5 - 45 34 - 155 622 - 5000
BW to a Central LaboratoryHousing One or More MajorExperiments
34 - 155 155 - 622 2500 - 10000
BW on a transoceanic Link 1.5 - 20 34 - 155 622 - 5000
US-CERN BW Requirements US-CERN BW Requirements Projection Projection (PRELIMINARY)(PRELIMINARY)
2001 2002 2003 2004 2005 2006
Installed LinkBW in MbpsIncl. New SLACThroughput [*]
310
(120)
622
(250)
1600
(400)
2400
(600)
4000
(1000)
6500 [#]
(1600)
[#] Includes ~1.5 Gbps Each for ATLAS and CMS, Plus Babar, Run2 and Other[*] D0 and CDF at Run2: Needs Presumed to Be to be Comparable to BaBar
Daily, Weekly, Monthly and Yearly Statistics Daily, Weekly, Monthly and Yearly Statistics on the 45 Mbps US-CERN Linkon the 45 Mbps US-CERN Link
HEP Network RequirementsHEP Network Requirementsand STARTAPand STARTAP
Beyond the requirement of adequate bandwidth, Beyond the requirement of adequate bandwidth, physicists in HENP’s major experiments depend on:physicists in HENP’s major experiments depend on:
Network and user software that will work together to Network and user software that will work together to provide high throughput and to manage the provide high throughput and to manage the bandwidth effectivelybandwidth effectively
A suite of videoconference and high-level tools for A suite of videoconference and high-level tools for remote collaboration that make data analysis from remote collaboration that make data analysis from the US (and from other world regions) effectivethe US (and from other world regions) effective
An integrated set of local, regional, national and An integrated set of local, regional, national and international networks that interoperate seamlessly, international networks that interoperate seamlessly, without bottleneckswithout bottlenecks
Configuration at Chicago with Configuration at Chicago with KPN/Qwest KPN/Qwest
HEP Network RequirementsHEP Network Requirementsand STARTAPand STARTAP
The STARTAP, a professionally managed international The STARTAP, a professionally managed international peering point with an open HP policy, has been and peering point with an open HP policy, has been and will continue to be vital for US involvement in the LHC, will continue to be vital for US involvement in the LHC, and thus for the progress of the LHC physics program.and thus for the progress of the LHC physics program.
Our development of worldwide Data Grid systems,Our development of worldwide Data Grid systems,in collaboration with the European Union and otherin collaboration with the European Union and otherworld regions, will depend on the STARTAP for jointworld regions, will depend on the STARTAP for jointprototyping, tests and developments using next-prototyping, tests and developments using next-generation network, software and database technology.generation network, software and database technology.
A scalable and cost-effective growth path for the A scalable and cost-effective growth path for the STARTAP will be needed, as a central component STARTAP will be needed, as a central component of international networks for HENP, and other fields.of international networks for HENP, and other fields.
An optical STARTAP handling OC-48 and OC-192An optical STARTAP handling OC-48 and OC-192links, with favorable peering and transit arrangements links, with favorable peering and transit arrangements across the US would be well-matched to our future plans.across the US would be well-matched to our future plans.
US-CERN line connection to Esnet:US-CERN line connection to Esnet:to HENP Labs Through STARTAPto HENP Labs Through STARTAP
TCP throughput performance: TCP throughput performance: Caltech/CERN Via STARTAPCaltech/CERN Via STARTAP
From Caltech to CERN
From CERN to Caltech
Vancouver
Calgary
Regina Winnipeg
Ottawa
Montreal
Toronto
Halifax
St. John’s
Fredericton
Charlottetown
Chicago
Seattle
New York
Los Angeles Miami
Europe
Dedicated Wavelength
or SONET channel
OBGP switches
Optional Layer 3 aggregation service
Large channel WDM system
CA*net 4 Possible ArchitectureCA*net 4 Possible Architecture
Pasadena
Intermediate ISP
Tier 1 ISPTier 2 ISP
AS 1 AS 2 AS 3 AS 4
AS 5
Dual Connected
Router to AS 5
Optical switch looks like BGP router and AS1 is direct connected to Tier 1 ISP but still transits AS 5
Router redirects networks with heavy traffic load to optical switch, but routing policy still maintained by ISP
Bulk of AS 1 traffic is to Tier 1 ISP
For simplicity only data forwarding
paths in one direction shown
Red Default Wavelength
OBGP Traffic Engineering - PhysicalOBGP Traffic Engineering - Physical
Worldwide Computing IssuesWorldwide Computing Issues
Beyond Grid Prototype Components: Integration of Beyond Grid Prototype Components: Integration of Grid Prototypes for End-to-end Data TransportGrid Prototypes for End-to-end Data Transport
Particle Physics Data Grid (PPDG) ReqMParticle Physics Data Grid (PPDG) ReqM PPDG/EU DataGrid GDMP for CMS HLT ProductionsPPDG/EU DataGrid GDMP for CMS HLT Productions
Start Building the Grid System(s): Integration with Start Building the Grid System(s): Integration with Experiment-specific software frameworksExperiment-specific software frameworks
Derivation of Strategies (MONARC Simulation System) Derivation of Strategies (MONARC Simulation System) Data caching, query estimation, co-schedulingData caching, query estimation, co-scheduling Load balancing and workload management amongst Load balancing and workload management amongst
Tier0/Tier1/Tier2 sites (SONN by Legrand)Tier0/Tier1/Tier2 sites (SONN by Legrand) Transaction robustness: simulate and verifyTransaction robustness: simulate and verify
Transparent Interfaces for Replica ManagementTransparent Interfaces for Replica Management Deep versus shallow copies: Thresholds; Deep versus shallow copies: Thresholds;
tracking, monitoring and controltracking, monitoring and control
VRVS Remote Collaboration VRVS Remote Collaboration System: StatisticsSystem: Statistics
0200400600800
1000120014001600180020002200240026002800300032003400
Ja
n-9
7F
eb
-97
Ma
r-9
7A
pr-
97
Ma
y-9
7J
un
-97
Ju
l-9
7A
ug
-97
Se
p-9
7O
ct-
97
No
v-9
7D
ec
-97
Ja
n-9
8F
eb
-98
Ma
r-9
8A
pr-
98
Ma
y-9
8J
un
-98
Ju
l-9
8A
ug
-98
Se
p-9
8O
ct-
98
No
v-9
8D
ec
-98
Ja
n-9
9F
eb
-99
Ma
r-9
9A
pr-
99
Ma
y-9
9J
un
-99
Ju
l-9
9A
ug
-99
Se
p-9
9O
ct-
99
No
v-9
9D
ec
-99
Ja
n-0
0F
eb
-00
Ma
r-0
0A
pr-
00
Ma
y-0
0J
un
-00
Ju
l-0
0A
ug
-00
Se
p-0
0
Months
Number of Machines and People registered in VRVS
Machines Registered People Registered
30 Reflectors52 Countries
Mbone, H.323, MPEG2Streaming, VNC
VRVS: Mbone/H.323/QT VRVS: Mbone/H.323/QT SnapshotSnapshot
VRVS Future evolution/integration
VRVS Future evolution/integration (R&D)(R&D)
Wider Deployment and Support of
Wider Deployment and Support of
VRVS.VRVS.High Quality video and audio (MPEG1,
High Quality video and audio (MPEG1,
MPEG2,..).MPEG2,..).Shared virtual workspaces, applications,
Shared virtual workspaces, applications,
and and environment environment
Integration of H.323 ITU Standard
Integration of H.323 ITU Standard
Quality of Service (QoS) over the
Quality of Service (QoS) over the
networknetwork
Improved security, authentication and
Improved security, authentication and confidentialityconfidentiality
Remote control of video cameras via a
Remote control of video cameras via a
Java appletJava applet
Demonstrations (HN, J. Bunn, Demonstrations (HN, J. Bunn, P. Galvez): CMSOO and VRVS P. Galvez): CMSOO and VRVS
CMSOO: CMSOO: Java 3D Java 3D Event Event
DisplayDisplayIGrid2000
Yokohama, July 2000
STARTAP: Selected HENP STARTAP: Selected HENP Success Stories (1)Success Stories (1)
Onset of large scale optimized Production file Onset of large scale optimized Production file transfers, involving both HENP Labs & Universitiestransfers, involving both HENP Labs & Universities Babar, CMS, ATLASBabar, CMS, ATLAS Upcoming D0, CDF at FNAL/Run2; RHICUpcoming D0, CDF at FNAL/Run2; RHIC
Seamless remote access to Object databasesSeamless remote access to Object databases CMSOO demos: IGrid2000 (Yokohama)CMSOO demos: IGrid2000 (Yokohama) Now starting on distributed CMS ORCA OO Now starting on distributed CMS ORCA OO
(TB to PB) DB Access(TB to PB) DB Access CMS User Analysis Environment (UAE)CMS User Analysis Environment (UAE)
Worldwide Grid-enabled view of the data, Worldwide Grid-enabled view of the data, along along with visualizations, data with visualizations, data
presentation presentation and analysisand analysis A User-view across the Data GridA User-view across the Data Grid
STARTAP: Selected HENP STARTAP: Selected HENP Success Stories (2)Success Stories (2)
A Principal testbed to develop production A Principal testbed to develop production Grid systems, of worldwide scope Grid systems, of worldwide scope Grid Data Management Prototype (GDMP; US/EU) Grid Data Management Prototype (GDMP; US/EU) GriPhyN: 18-20 University facilities serving GriPhyN: 18-20 University facilities serving
CMS, ATLAS,LIGO and SDSS, CMS, ATLAS,LIGO and SDSS, Built on a strong foundation of grid security and Built on a strong foundation of grid security and
information infrastructure Foundationinformation infrastructure Foundation Deploying a Grid Virtual Data Toolkit (VDT)Deploying a Grid Virtual Data Toolkit (VDT)
VRVS: Worldwide-extensible videoconferencingVRVS: Worldwide-extensible videoconferencing and shared virtual spaces and shared virtual spaces
Future: Forward-looking view of Mobile Agent Future: Forward-looking view of Mobile Agent Coordination Architectures Coordination Architectures Survivable Loosely Coupled Systems withSurvivable Loosely Coupled Systems with
Unprecedented Scalability Unprecedented Scalability