IHEP Computing Center Site IHEP Computing Center Site ReportReport

Shi, Jingyan Shi, Jingyan (shi.jingyan@ihep.ac.cn)(shi.jingyan@ihep.ac.cn)Computing Center,Computing Center,IHEPIHEP

Shi,Jingyan/CC/IHEP 23/4/20 - 2

IHEP Brief IHEP Brief introductionintroduction

• The largest(~1000 stuffs) fundamental The largest(~1000 stuffs) fundamental research center in China with research fieldsresearch center in China with research fields i includncluding: ing: • Particle Physics ExperimentsParticle Physics Experiments

• Cosmic Ray/Astrophysics experimentsCosmic Ray/Astrophysics experiments

• Theoretical PhysicsTheoretical Physics

• ……

• Scientific projects:Scientific projects:• BESIII experiment running on BEPC BESIII experiment running on BEPC • ARGO-YBJ experiment ARGO-YBJ experiment • Daya Bay Daya Bay rreactoreactor n neutrino eutrino eexperiment  xperiment  • ATLAS, CMS experiment on LHCATLAS, CMS experiment on LHC• AMS, HMXT …AMS, HMXT …

Beijing Spectrum ARGO-YBJ Detector

CC-IHEP at a CC-IHEP at a GlanceGlance

• The Computing Center was created in The Computing Center was created in 1980’s1980’s• Provided computing service to BES, the Provided computing service to BES, the

experiment on BEPCexperiment on BEPC

• Rebuilt in 2005 for the new projects:Rebuilt in 2005 for the new projects:• BES-III on BEPC-IIBES-III on BEPC-II• Tier-2 for ATLAS, CMSTier-2 for ATLAS, CMS• Cosmic ray experiments Cosmic ray experiments

• 35 FTEs, half of them for computing 35 FTEs, half of them for computing facilityfacility

Shi,Jingyan 23/4/20 - 3

Shi,Jingyan 23/4/20 - 4

10Gbit Ethernet

3+ PB disk storage ( Lustre ,dpm, dCache)

More than 8000 CPU cores

5PB tape storage

Login, monitoring, scheduling,AFS,backup…

Outside data farms:LHC, YBJ,DAYABAY

Local data farm

WAN

IHEP Campus IHEP Campus NetworkNetwork

• Network upgrade finished last monthNetwork upgrade finished last month• IPv4/IPv6 enabled for all usersIPv4/IPv6 enabled for all users• Campus wireless network covered Campus wireless network covered

Shi,Jingyan– Kan, Bowen/CC/IHEP 23/4/20 - 5

Shi,Jingyan/CC/IHEP 23/4/20 - 6

Network connectionNetwork connection

DayaBay

ShenZhen

BeijingCSTNet

HongKong

IHEP

USA

GLORIAD 10G

ASGC

TEIN3

IPv4 10G IPv6

BeijingTsinghua

YBJ

EUR.

2.5G 1G

155M

155M

2.5G

Others

EDU.CN2.5G

Computing Computing ResourcesResources

• Local Cluster & Grid SiteLocal Cluster & Grid Site• Cpu:Cpu:

• Work node: More than 8000 cpu/coresWork node: More than 8000 cpu/cores

• Storage:Storage:• 3+PB storage3+PB storage

Lustre for local clusterLustre for local cluster dCache for CMSdCache for CMS DPM for ATLASDPM for ATLAS

• 5PB Tape storage5PB Tape storage

• SchedulerScheduler• Cluster: PBS + MauiCluster: PBS + Maui

Shi,Jingyan 23/4/20 - 7

File system - Lustre

• 4 MDSs, 39 OSSs, 376 OSTs, 800 client nodes, 135 million files

• Lustre Version: 1.8.5 ( upgraded in July) and 1.8.6 (with a new mount point)

• Capacity: 2.8PB • IHEP is considering binding Lustre with

CASTOR 1.7 using the HSM function provided by Lustre 2.x, however, it seems that the HSM feature has not totally landed in stable version of Lustre

Disk ErrorDisk Error• More than 3000 pieces of disks are used for More than 3000 pieces of disks are used for

the storage and 2/3 have expired guarantee the storage and 2/3 have expired guarantee periodperiod• 1TB disks: 1972 pieces1TB disks: 1972 pieces• 2TB disks: 752 pieces2TB disks: 752 pieces• 3TB disks: 432 pieces3TB disks: 432 pieces

• Disk hard error happened once 3-4 days in Disk hard error happened once 3-4 days in averageaverage

• Spare parts are limited, especially for 1TB Spare parts are limited, especially for 1TB disksdisks

• Considering a way to upgrade thoseConsidering a way to upgrade those disks disks smoothlysmoothly

Shi,Jingyan 23/4/20 - 9

Shi,Jingyan– Kan, Bowen/CC/IHEP 23/4/20 - 10

IHEP - CAIHEP - CA• Number of issued certificatesNumber of issued certificates

Last update: Apr20 2012 UTC+8

User Certificate

Host Certificate

Service Certificate

Total

VALID 112 100 0 212

Revoked 254 241 1 496

EXPIRED 344 307 6 657

IHEP PKU HKU BUAA NJNU SEU RSGS

CNIC TSINGHUA

CCNU SDU NNU NJU USTC

• Cooperation OrganizationsCooperation Organizations

CA need to be CA need to be upgradedupgraded

• According to the GFD.125, emailAddress (or According to the GFD.125, emailAddress (or

Email) now MUST NOT be used in subject Email) now MUST NOT be used in subject

or issuer DNs. or issuer DNs.

• IHEP will follow the steps below to remove the IHEP will follow the steps below to remove the

emailAddress.emailAddress.1. Modify the CP/CPS to comply with GFD.1251. Modify the CP/CPS to comply with GFD.125

2. Generate a new key pair (new root CA cert)2. Generate a new key pair (new root CA cert) . Register it to the . Register it to the

IGTF repository so that the new root CA cert and related files will IGTF repository so that the new root CA cert and related files will

be included in the IGTF CA distributionbe included in the IGTF CA distribution

3. 3. Inform the relying parties that the subject DN of CA certs and Inform the relying parties that the subject DN of CA certs and

EE certs would be changed. NEE certs would be changed. New root CA will issue new EE certsew root CA will issue new EE certs

4. 4. Revoke the old CA certificate until all EE certificates issued by Revoke the old CA certificate until all EE certificates issued by

the old CA key will be expiredthe old CA key will be expired

Shi,Jingyan 23/4/20 - 11

BEIJING-LCG2 Site reportBEIJING-LCG2 Site report

BEIJING-LCG2 Site report

Reliability and Availability

Site Operation

• Some new computing resources will be added• 16 old WN will be replaced with new blade

before JUL. 2012• HEPSPEC 2006 will rise up from 8000 to

9600

• EGEE to EGI migration• Testing new system• Grid middleware upgrade• Keep eyes on other sites’ migration.

Question & Thank You