Download - IHEP(Beijing LCG2) Site Report
IHEP(Beijing LCG2) Site ReportIHEP(Beijing LCG2) Site Report
Fazhi.Qi, Gang ChenFazhi.Qi, Gang ChenComputing Center,IHEPComputing Center,IHEP
OutlineOutline
• InfrastructureInfrastructure
• Local Cluster StatusLocal Cluster Status
• LCG Tire2 Site StatisticsLCG Tire2 Site Statistics
•Management & Operation Management & Operation
• SummarySummary
Chen Gang/CC/IHEP 23/4/20 - 2
InfrastructureInfrastructure
• Serving more than 1000 usersServing more than 1000 users
• Power supply capacity: 1800Kw Power supply capacity: 1800Kw
Shi,Jingyan/CC/IHEP 23/4/20 3
• Cooling: water Cooling: water cooling rack for cooling rack for the blade serversthe blade servers
• Water cooling rackWater cooling rack
• Inter-row air conditioningInter-row air conditioning
• Cooling capacity per rack: Cooling capacity per rack:
28kw28kw
Infrastructure UpgradeInfrastructure Upgrade
Shi,Jingyan CC--IHEP 23/4/20 - 4
Power Capacity: 1800kwPower Capacity: 1800kw
Cooling SystemCooling System
Chen Gang/CC/IHEP 23/4/20 5
Local Cluster --Computing Local Cluster --Computing NodesNodes
• Most for BES,YBJ,DYB,Atlas,CMS Most for BES,YBJ,DYB,Atlas,CMS
experimentsexperiments
• Some small projects addedSome small projects added
• Blade system IBM/HP/DellBlade system IBM/HP/Dell
• Blade links with GigE/IBBlade links with GigE/IB
• Chassis links to central switch with 10GigEChassis links to central switch with 10GigE
• 886 computing nodes: 7082 CPU-886 computing nodes: 7082 CPU-
corescores
• Most running SL5.5 (64 bit)Most running SL5.5 (64 bit)
• Intend to migrate to SL5.8Intend to migrate to SL5.8
• A small part stayes in running A small part stayes in running
SL4.5 (32 bit)SL4.5 (32 bit)
• Torque: 2.5.5Torque: 2.5.5
• Maui: 3.2.6Maui: 3.2.6
• Intend to upgrade to 3.4.4 Intend to upgrade to 3.4.4
or higher to support MPI or higher to support MPI
jobsjobs
• Tools developed to Tools developed to
monitor the resources monitor the resources
usage, queue status usage, queue status
etc.etc.
• Accounting tool Accounting tool
developeddeveloped
Chen Gang/CC/IHEP 23/4/20 - 6
Local Cluster -- Local Cluster -- SchedulerScheduler
Scheduler
• 50 queues to fit various requests
• Besides serial jobs, MPI, GPU jobs are also supported
• Testbed
• Integration of Torque and openstack
• Managing and scheduling VM nodes in batch-cloud
Chen.Gang/CC/IHEP 23/4/20 7
Local Cluster -- StorageLocal Cluster -- Storage
• Gluster system installed Gluster system installed
• Storage provided less than 4 months Storage provided less than 4 months
• Keeps optimizing performanceKeeps optimizing performance
• Adjust to deal with the new bugsAdjust to deal with the new bugs
• Total space: 153TB, Used space: 145TBTotal space: 153TB, Used space: 145TB
Chen Gang/CC--IHEP 23/4/20 - 8
Beijing LCG Tier II SiteBeijing LCG Tier II Site
• For CMS, ATLAS experimentsFor CMS, ATLAS experiments
• 1000+ Job slots1000+ Job slots
• Storage: Storage:
• 320TB dCache 320TB dCache
• 320TB dpm 320TB dpm
• 1T disks were replaced by 2T disks1T disks were replaced by 2T disks
Chen Gang/CC--IHEP 23/4/20 - 9
Beijing LCG Tier II SiteBeijing LCG Tier II Site
• CPU TimeCPU Time
Orient+
Network ConnectionNetwork Connection
DayaBay
BeijingCSTNet
HongKong
IHEP
USA
GLORIAD 10GASGC
IPv4 10G IPv6
BeijingTsinghua
YBJ
EUR.
2.5G
155M
155M
10G
Others
EDU.CN10G
Chen Gang/CC--IHEP 23/4/20 - 11
Perfsonar @IHEPPerfsonar @IHEP• Two hosts for perfsonarTwo hosts for perfsonar
• Perfsonar.ihep.ac.cn for Perfsonar.ihep.ac.cn for
Bandwidth testBandwidth test
• Perfsonar2.ihep.ac.cn for Perfsonar2.ihep.ac.cn for
Latency testLatency test
• Network performance Network performance
tuning is in progress tuning is in progress
between IHEP and EU. Sitesbetween IHEP and EU. Sites
• http://twiki.ihep.ac.cn/twiki/http://twiki.ihep.ac.cn/twiki/
bin/view/bin/view/
InternationalConnectivity/InternationalConnectivity/
IHEP-CCIN2P3IHEP-CCIN2P3
Chen Gang/CC--IHEP 23/4/20 - 12
Network Research (SDN@IHEP)Network Research (SDN@IHEP)• GoalGoal
• A flexible, reliable and high performance HEP data transfer A flexible, reliable and high performance HEP data transfer network (virtual and private) and system platform in Chinanetwork (virtual and private) and system platform in China
• IPv4 and IPv6 supportedIPv4 and IPv6 supported• The traffic can be switched between IPv4 and IPv6 The traffic can be switched between IPv4 and IPv6
infrastructure and physical path automatically or manually infrastructure and physical path automatically or manually based the network performance and applicationsbased the network performance and applications
• SDN@IHEP SDN@IHEP IHEPDTN IHEPDTN
• End user networkEnd user network• Backbone networkBackbone network (( IPv6 & IPv4IPv6 & IPv4 ))• SDN Switch (L2VPN gateway & Openflow supported)SDN Switch (L2VPN gateway & Openflow supported)• Control center (API to Application)Control center (API to Application)• Applications(FTS/NMS/…….)Applications(FTS/NMS/…….)
• MembersMembers
• IHEP/SJU/SDU/TsingHua/……IHEP/SJU/SDU/TsingHua/……• Network VendorNetwork Vendor :: Ruijie NetworksRuijie Networks
SDN@IHEP modelSDN@IHEP model
• Most part of computing environment running Most part of computing environment running
wellwell
• New gLuster system is in productionNew gLuster system is in production
• Network performance between IHEP-Eur. got Network performance between IHEP-Eur. got
an clear improvement an clear improvement
• New Management and Operation system will New Management and Operation system will
be deployed to improve the efficiencybe deployed to improve the efficiency
SummarySummary
Chen Gang/CC/IHEP 23/4/20 - 15
Thank you!Thank you!
Questions?Questions?
Chen Gang/CC/IHEP 23/4/20 - 16