Download - PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015
![Page 1: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/1.jpg)
PANDA: Networking Update
Kaushik DeUniv. of Texas at Arlington
SC15 DemoNovember 18, 2015
![Page 2: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/2.jpg)
Overview
PanDA workload management system was developed for the ATLAS experiment at the Large Hadron Collider Hundreds of petabytes of data per year, thousands of users
worldwide, many dozens of complex applications… Leading to ~500 scientific publications Discovery of the Higgs boson, search for dark matter…
A new approach to distributed computing A huge hierarchy of computing centers working together Main challenge – how to provide efficient automated performance Auxiliary challenge – make resources easily accessible to all users
Network Aware Data Management is crucial for systems like PanDA to work efficiently
November 18, 2015Kaushik De 2
![Page 3: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/3.jpg)
The ATLAS Experiment at the LHC
November 18, 2015Kaushik De 3
Largest data sensor in the world
![Page 4: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/4.jpg)
Kaushik De 4
Business emails sent3000PB/year
(Not managed asa coherent data set)
Google search100PB
Facebook uploads180PB/year
KaiserPermanente
30PB
LHC data15PB/yr
YouTube15PB/yr
USCensus
Lib ofCongress
ClimateDB
Nasdaq
Wired 4/2013
Current ATLAS data set, all data products:
160 PB
1+M files transferred per dayhttp://www.wired.com/magazine/2013/04/bigdata/
Big Data in 2013
ATLAS is Big Data
November 18, 2015
ATLASdata
![Page 5: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/5.jpg)
Distributed Computing in ATLAS
Workload ManagemeentSystem
November 18, 2015Kaushik De 5
![Page 6: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/6.jpg)
Paradigm Shift in HEP Computing
New Ideas from PanDA Distributed resources are
seamlessly integrated All users have access to
resources worldwide through a single submission system
Uniform fair share, priorities and policies allow efficient management of resources
Automation, error handling, and other features in PanDA improve user experience
All users have access to same resources
Old HEP paradigm Distributed resources are
independent entities Groups of users utilize
specific resources (whether locally or remotely)
Fair shares, priorities and policies are managed locally, for each resource
Uneven user experience at different sites, based on local support and experience
Privileged users have access to special resources
November 18, 2015Kaushik De 6
![Page 7: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/7.jpg)
PanDA Scale
Current scale – 35M jobs completed every month at >hundred sitesFirst exascale system in HEP – 1.2 Exabytes processed in 2013
November 18, 2015Kaushik De 7
![Page 8: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/8.jpg)
CPU Consumption
During LHC Run 1+2, per month, aggregated by federationNovember 18, 2015Kaushik De 8
![Page 9: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/9.jpg)
The Growing PanDA EcoSystem
ATLAS PanDA core US ATLAS, CERN, UK, DE, ND, CA, Russia, OSG …
ASCR/HEP BigPanDA DoE funded project at BNL, UTA – PanDA beyond HEP, at LCF
CC-NIE ANSE PanDA NSF funded network project - CalTech, Michigan, Vanderbilt, UTA
HPC and Cloud PanDA – very active Taiwan PanDA – AMS and other communities Russian NRC KI PanDA, JINR PanDA – ATLAS,
COMPASS, ALICE, NICA, Biology… AliEn PanDA, LSST PanDA, other experiments
November 18, 2015Kaushik De 9
![Page 10: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/10.jpg)
Resources Accessible via PanDA
ManyOthers
About 200,000 job slots used continuously 24x7x365
November 18, 2015Kaushik De 10
![Page 11: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/11.jpg)
PanDA References
PanDA – Production and Distributed Analysis System Deployed on WLCG infrastructure Standards based implementation
REST framework – HTTP/S Oracle or MySQL backends CondorG based pilot factories Python packages available from SVN and GitHub Command-line and GUI/Web interfaces
Reference https://twiki.cern.ch/twiki/bin/view/PanDA/PanDA http://pandawms.org/
November 18, 2015Kaushik De 11
![Page 12: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/12.jpg)
PanDA Networking Projects
DOE ASCR and HEP funded project “Next Generation
Workload Management and Analysis System for Big Data” Generalization of PanDA for HEP and other data-intensive sciences
Project participants from ANL, BNL, UT Arlington
WP3 (Leveraging intelligent networks): Integrating
network services and real-time data access to the
PanDA workflow ANSE project – funded by NSF CC*NIE
CalTech, Michigan, Vanderbilt, UT Arlington Advanced Network Services for Experiments
November 18, 2015Kaushik De 12
![Page 13: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/13.jpg)
WP3 PanDA and Networking
PanDA as workload manager PanDA automatically chooses job execution site
Multi-level decision tree – task brokerage, job brokerage, dispatcher Also predictive workflows – like PD2P (PanDA Dynamic Data Placement)
Site selection is based on processing and storage requirements Why not use network information in this decision? Can we go even further – network provisioning?
Network knowledge useful for all phases of job cycle
Network as resource Optimal site selection should take network capability into account
We do this already – but indirectly using job completion metrics
Network as a resource should be managed (i.e. provisioning) We also do this crudely – mostly through timeouts, self throttling
Goal for PanDA Direct integration of networking with PanDA workflow – never attempted before
for large scale automated WMS systemsNovember 18, 2015Kaushik De 13
![Page 14: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/14.jpg)
Job Workflow
November 16, 2015Kaushik De 14
Panda jobs go through a succession of steps tracked in central DB Defined Waiting Assigned Throttled Activated Sent
Starting Running Holding Transferring Finished/failed Cancelled
![Page 15: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/15.jpg)
Using Network Information
Pick a few use cases Cases which are important to PanDA users Enhance workload management through use of network
Case 1: Improve User Analysis workflow Case 2: Improve Tier 1 to Tier 2 workflow Step by step approach
Collect network information Storage and access Using network information Using dynamic circuits
November 18, 2015Kaushik De 15
![Page 16: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/16.jpg)
Sources of Network Information
DDM Sonar measurements Actual transfer rates for files between all sites (Tier 1 and Tier 2) This information is normally used for site white/blacklisting Measurements available for small, medium, and large files
perfSonar (PS) measurements perfSonar provides dedicated network monitoring data All WLCG sites are being instrumented with PS boxes US sites are already instrumented and monitored
Federated XRootD (FAX) measurements Read-time of remote files are measured for pairs of sites
This is not an exclusive list – just a starting point
November 18, 2015Kaushik De 16
![Page 17: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/17.jpg)
Network Data Repositories
Native data repositories Historical data stored from collectors SSB – site status board for sonar and perfSonar data FAX data is kept independently and uploaded
AGIS (ATLAS Grid Information System) Most recent / processed data only – updated periodically Mixture of push/pull – depending on source of data
schedConfigDB Internal Oracle DB used by PanDA for fast access Uses standard ATLAS collector
November 18, 2015Kaushik De 17
![Page 18: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/18.jpg)
November 18, 2015Kaushik De 18
![Page 19: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/19.jpg)
Case 1: Faster User analysis
First use case for network integration with PanDA Goal - reduce waiting time for user jobs
User analysis jobs normally go to sites with local input data This can occasionally lead to long wait times (jobs are re-brokered if
possible, or PD2P data caching will make more copies eventually to reduce congestion)
While nearby sites with good network access may be idle
Brokerage uses concept of ‘nearby’ sites Use cost metric generated with Hammercloud tests Calculate weight based on usual brokerage criteria (availability of CPU
resources, data location, release…) plus new network transfer cost Jobs will be sent to the site with best overall weight
Throttling is used to manage load on network
November 18, 2015Kaushik De 19
![Page 20: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/20.jpg)
November 18, 2015Kaushik De 20
![Page 21: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/21.jpg)
Measure of Success
November 18, 2015Kaushik De 21
Many sites used for overflow
Failure rate is manageable
![Page 22: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/22.jpg)
First Tests
Tested in production for ~1 day in March, 2014 Useful for debugging and tuning direct access infrastructure We got first results on network aware brokerage
Job distribution 4748 jobs from 20 user tasks which required data from congested
U.S. Tier 1 site were automatically brokered to U.S. Tier 1/2 sites
120417 555
837
408660366
558
123030
41730
30128
30 30 30 30 30
Number of Jobs per Task
November 18, 2015Kaushik De 22
![Page 23: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/23.jpg)
Brokerage Results
November 18, 2015Kaushik De 23
553 566 568 569 570 571 573 574 598 605 615 617 622 640 647 655 662 665 668 6811
10
100
1000
10000
FAX/non-FAX Ratio
# of Local Jobs
# of Remote Jobs
Task Number
553 566 568 569 570 571 573 574 598 605 615 617 622 640 647 655 662 665 668 6810
100
200
300
400
500
600
700
Job Wait Times
Local Jobs Wait Time
Remote Jobs Wait Time
Task Number
![Page 24: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/24.jpg)
Early Example from Oct, 2014
How do we measure success?Let’s look at example task.
November 18, 2015Kaushik De 24
ThisYear
![Page 25: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/25.jpg)
Jobs from task 4199733 on oct 3
775675
3080
17
520
1964
3597
13
4368
4136
BU:BNLBU:MWT2BNLBUMWT2MWT2DESYHUINFNMWT2
November 18, 2015Kaushik De 25
![Page 26: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/26.jpg)
Job wait times for example task
BU:BNL BU:MWT2 BNL BU MWT2 MWT2 DESY HU INFN MWT2
0
20
40
60
80
100
120
140
160
Wait TimeCPU Time Used
November 18, 2015Kaushik De 26
![Page 27: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/27.jpg)
Conclusions for Case 1
Network data collection working well Additional algorithms to combine network data will be tried HC tests working well – PS data not robust yet
PanDA brokerage worked well Achieved goal of reducing wait time – though anecdotally Well balanced local vs remote access Need fine tuning – we have a lot of data now
We have overflow working for FAX We need deeper study to optimize
November 18, 2015Kaushik De 27
![Page 28: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/28.jpg)
Case 2: Cloud Selection
Second use case for network integration with PanDA Optimize choice of T1-T2 pairings (cloud selection)
In ATLAS, production tasks are assigned to Tier 1’s Tier 2’s are attached to a Tier 1 cloud for data processing Any T2 may be attached to multiple T1’s Currently, operations team makes this assignment manually This could/should be automated using network information For example, each T2 could be assigned to a native cloud by
operations team, and PanDA will assign to other clouds based on network performance metrics
November 18, 2015Kaushik De 28
![Page 29: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/29.jpg)
DDM Sonar Data
http://aipanda021.cern.ch/networking/t1tot2d_matrix/
November 18, 2015Kaushik De 29
![Page 30: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/30.jpg)
Tier 1 View
November 18, 2015Kaushik De 30
![Page 31: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/31.jpg)
Tier 2 View
November 18, 2015Kaushik De 31
![Page 32: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/32.jpg)
Improving Site Association
November 18, 2015Kaushik De 32
![Page 33: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/33.jpg)
Conclusion for Case 2
Worked well as a demonstrator Migrating to concept of “World Cloud”
Any site can be potentially “Nuclues” for data aggregation Any other site can provide processing Removes Tier1-2 hierarchy Network integration is crucial for this new implementation
November 18, 2015Kaushik De 33
![Page 34: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/34.jpg)
Operationalizing perfSONAR
WLCG has formed Working Group for LHC experiments Network and Transfer Metrics WG – chaired by Marian Babik
& Shawn McKee https://twiki.cern.ch/twiki/bin/view/LCG/NetworkTransferMetrics Mandate
Ensure all relevant network and transfer metrics are identified, collected and published
Ensure sites and experiments can better understand and fix networking issues
Enable use of network-aware tools to improve transfer efficiency and optimize experiment workflows
Report 05/11/2015 perfSONAR collector, datastore, publisher and dashboard now in
production (stable operations)
November 18, 2015Kaushik De 34
![Page 35: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/35.jpg)
From Marian Babik
November 18, 2015Kaushik De 35
![Page 36: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/36.jpg)
From Marian Babik
November 18, 2015Kaushik De 36
![Page 37: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/37.jpg)
From Marian Babik
November 18, 2015Kaushik De 37
![Page 38: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/38.jpg)
Conclusion
Rich area for future R&D Moving from “timeouts” to “metrics” based approach Direct integration of network information in decision making Next step – direct interaction with network elements
November 18, 2015Kaushik De 38
![Page 39: PANDA: Networking Update Kaushik De Univ. of Texas at Arlington SC15 Demo November 18, 2015](https://reader034.vdocuments.site/reader034/viewer/2022051401/5697c02a1a28abf838cd81fe/html5/thumbnails/39.jpg)
Live Pages
http://dashb-atlas-job.cern.ch/dashboard/request.py/dailysummary
http://aipanda021.cern.ch/networking/t1tot2d_matrix/ http://aipanda021.cern.ch/networking/CurrentHomeForeign
T2SitesPerCloud/