gungwon kang & jiwoong kim (kisti) ligo data grid and kisti june 27, 2015 at 8 th j-k joint...
TRANSCRIPT
Gungwon Kang & Jiwoong Kim (KISTI)
LIGO Data Grid and KISTI
June 27, 2015 at 8th J-K Joint Workshop on KAGRA, Gwangju in Korea
2
OUTLINE
I. KISTI GSDC Overview
II. KISTI LDG (LIGO Data Grid)
III. Conclusion
3
• National research institute for information technology since 1962• About 600 people working for Supercomputing & Networking and
National Information Service (development & analysis)
• Running High-Performance Computing Facility• Total 3,398 Nodes (30,592 CPUs, 360 TFlops at peak), 1,667 TB
storage (introduced from 2008)
KISTI: Korea Institute of Science and Technology Information
Intel Xeon X5570 2.93GHz (Nehalem)
Rpeak: 300TFCPU: 25,408Memory: 76.8TBStorage: 1,061TB
I. KISTI GSDC Overview
4
GSDC: Global Science experiment Data hub Center
• National project to promote data-based science research experiments by pro-viding computing and storage resources: HEP and other fields
• Running Data-Intensive Computing Facility• ~20 Staffs: system administration, experiment support, external-relations, administra-
tion and students• CPU: ~5,900 cores. Storage: ~6.8PB• Budget: ~ 6M$/year
• Supporting experiments: ALICE, CMS, Belle, LIGO, RENO, Genomic Medicine, etc.
GSDC Facility
HP Servers
Hitachi VSP Storage
National Institute of Supercomputing and Networking 2013. 10 . 28.
CPU (cores)
2014.12
Field / Experiment
Particle/Nu-clear Physics
CERN (ALICE) 2,520 ~2,000
KiAF 108 100
CERN (T3) 240 110
KEK 224 110
BNL (STAR) 540 110
RENO 280 252
HCP 346 180
Astrophysics LIGO 420 152
Medical Sci-ence
Genom 144 150
G-brain 208 52
Meteorology PCMI 0 380
Etc. 870 126
Total 5,900 3,722
5
Storage (TB)
Resource allocations
6
Model Physical size Usable size
NetApp FAS2050
(SAN only, RAID6)104TB 50TB
NetApp FAS6080
(SAN & NAS, RAID6)334TB 200TB
Hitachi USP-V
(SAN & NAS, RAID6)960TB 600TB
EMC CX4-960C(SAN, RAID6) 1,920TB 1,250TB
EMC Isilon 108NL 1,620TB 1,400TB
Hitach VSP 1
(SAN, NAS, RAID6)
2013 758TB 500TB
2014 320TB 214TB
Hitach VSP 2
(SAN, NAS, RAID6)857TB 570TB
Total 6,873TB 4,784TB
GSDC CPU/Storage(2014.12.30)
구 분 내역
Model Hitach VSP, HNAS4080(4Node)
Disk Usable 700TB
RAID RAID6(6D+2P)
Cache Mem-
ory512GB
Front-end In-
terface8Gbps FC 32 ports
구 분 세 부 사 항 수 량
Com-
puting
Server
(Pur-
chase
d in
2014)
☐ 모델명 : HP DL360G8(1U)
☐ 사양 :
- E5-2680v2 2.8GHz
* 2P(20core)
- 128GB DDR3 1600Mhz SDRAM
- 600GB 10K SAS * 4EA
- 1GbE 2Port, 10GbE 2Port
- 8G HBA
- Redundant Power Supply
1,100
Core
(10-
Core
x 55
Nodes
)
7
II. LIGO Data Grid (LDG)
+ 8KISTI LDG T3
Result AnalysisDiscussion
User Authentication based on GSI
Storage
155 TBs
Tier1/2/3
LIGO / VIRGO Data
576 cores (780)wn3076~3110.sdfarm.kr (Condor)
Cluster
Login Node Web Server
KGWG F2F Meeting 8
ui04.sdfarm.kr (Condor)
ldas.ligo.kisti.re.kr
lgm.sdfarm.kr (Intel-Compiler-License)
ce04.sdfarm.kr (Condor)
ldr.sdfarm.kr(GridFTP server)
9
ui04.sdfarm.kr{Condor}
wn3076~3110.sdfarm.kr{Condor}
Central storage150TB
ldr.ligo.caltech.edu
….
ldr.sdfarm.kr{GridFTP server}
ldas.ligo.kisti.re.kr{web publication}
Connection/Job Submission
lgm.sdfarm.kr{Intel-Compiler-License}
ui test{Condor}
• System Configuration in more detail:
+KISTI LDG Resources (2015) 10
KGWG F2F Meeting
• Computation Resources(Worker Node)• 48 Node : 780cores (Hyperthread-17node) / 576 cores (Physical)
• Storage Resources(Only Data - /data/ligo/archive)• 155 TB (Expandable to 200 TB)
Size Used Avail Use (%)
/data/ligo/home 786GB 471GB 315GB 60
/data/ligo/lib 100GB 71GB 20GB 71
/data/ligo/scratch 4.73TB 3.96TB 784G 84
/data/ligo/archive
150TB 123TB 28TB 82
Total 155TB 127TB 29TB 8210
Cores RAM
Worker Node 780 / 576 72GB(Hyperthread)48GB(W/O
Hyperthread)
UI,CE,LGM,LDAS,LDR
60(12core per server) 24GB
Total 840 / 636
+Stored Data
- hoft frame: LIGO S5~6. Virgo VSR1~4
- RDS L1 frame: LIGO S6
11
+KISTI LDG Usage: 12
KGWG F2F Meeting12
기간 1 월 2 월 3 월 4 월 5 월 6 월 7 월 8 월 9 월 10 월 11 월 12 월 합계
2011
건수( 컴퓨팅 시간 , 일 )
124(81)
362(309)
80(171)
70(726)
44(500)
30(4)
187(193)
128(159)
1,025(2,143
)
2012
62 55 22 10 19 30 30 1 0 0 ? ? 229
2013
1,287 1,951 927 268 1 9 5,430 1,129 26 ? ? ?11,02
82014
331 918 921 703 367 881 1214 545 120 124 ? ? 6,124
2015
9,335 21,683 4,257 109 1 1435,40
0
• Work Node Usage: CPU 384 cores - 2014.11~2015.02: Max 78.9%, Ave 13% - 2015.01~2015.02: Max 60%, Ave 35% (※ Monitored by KISTI)
* Job 건수 만이 아닌 새로운 모니터링 Measure 필요 ( 예 , 2011 년 6 월 , 8 월 비교 )
13
• KGWG Korean Gravitational Wave Group (2008~): • ~30 people working in 8 universities and 3 government-funded institutes
• LIGO-Virgo and KAGRA
User groups (1)
KGWG
한국중력파연구협력단
소속 이름서울대 이형목 (PI) NIMS 오정근연세대 김정리 오상훈한양대 이현규 손재주
김경민 김환선이철훈 추형석
서강대 조규만 KISTI 강궁원부산대 이창환 장행진
김영민 김지웅김명국 윤희준
인제대 이형원 조희석김정초 KAERI 차용호
고려대 윤태현 GIST 강훈수조동현 경북대 박명구
명지대 김재완 군산대 김상표
+
Parameter estimation:
Chunglee Kim(Yonsei U/ KISTI GSDC), Hyungwon Lee, Chungcho Kim (Inje U)
+ LSC collaborators (Caltech, NU, UWM, Monclair State Univ.)
+ KAGRA collaborators (Osaka University)
User groups (2)14
CBC Signal and Noise Identification:
오정근 , 오상훈 , 손재주 , 김환선 , 추형석 (NIMS), 이창환 , 김영민 ( 부산대 )
※ KISTI 연구원 : 강궁원 , 장행진 , 김지웅 , 윤희준 , 조희석 (KISTI-GSDC)
iDQ pipeline 의 개선 (Deep Learning) 및 인공신경망 모듈 이식 Bank chisq 를 이용한 detection statistic 의 개선연구 중력파 채널과 보조채널간의 Correlation Analysis (CAGMon) HHT 를 이용한 Trigger generation 연구
- Mostly used by domestic researchers!
중력파 분석을 통한 모수추정 (parameter estimation) 보다 천체물리학적으로 “실제와 가깝고” , 계산적으로 효율적인 중력파형 개발
+User groups (3)
이름 소속 e-mail 계정
Kazuhiro
Hayama
Osaka
Tatsusya
Narikawa
Osaka
Hideyuki
Tagoshi
Osaka
Koh UenoOsaka
Hirotaka
Yuzurihara
Osaka
University
cu.ac.jp…
+이름 접속기록
Kazuhiro Hayama -
Tatsusya Narikawa
(588 분 )
narikawa pts/7 :pts/8:S.0 Wed Jun 3 16:49 - 16:51 (00:01)
narikawa pts/8 pascal.hep.osaka Wed Jun 3 16:49 - 16:51 (00:01)
narikawa pts/7 :pts/6:S.0 Thu May 21 12:12 - 12:43 (00:30)
narikawa pts/7 :pts/6:S.0 Thu May 21 11:20 - 11:51 (00:30)
narikawa pts/6 pascal.hep.osaka Thu May 21 11:20 - 12:45 (01:24)
narikawa pts/7 :pts/6:S.0 Thu May 21 10:52 - 10:52 (00:00)
Tue May 12 18:07 - 18:11 (00:04)
narikawa pts/18 :pts/14:S.0 Tue May 12 17:48 - 17:57 (00:09)
narikawa pts/14 pascal.hep.osaka Tue May 12 17:48 - 17:57 (00:09)
narikawa pts/19 :pts/16:S.0 Mon Apr 27 17:29 - 18:02 (00:32)
narikawa pts/19 :pts/16:S.0 Mon Apr 27 15:50 - 16:54 (01:04)
narikawa pts/16 pascal.hep.osaka Mon Apr 27 15:50 - 18:02 (02:11)
narikawa pts/19 :pts/16:S.0 Mon Apr 27 14:59 - 14:59 (00:00)
narikawa pts/19 :pts/16:S.0 Mon Apr 27 12:38 - 13:10 (00:31)
narikawa pts/16 pascal.hep.osaka Mon Apr 27 12:38 - 14:59 (02:20)
narikawa pts/5 :pts/1:S.0 Thu Apr 23 17:59 - 18:04 (00:05)
narikawa pts/1 pascal.hep.osaka Thu Apr 23 17:59 - 18:04 (00:05)
narikawa pts/5 :pts/1:S.0 Thu Apr 23 17:56 - 17:56 (00:00)
narikawa pts/1 pascal.hep.osaka Thu Apr 23 17:55 - 17:56 (00:00)
narikawa pts/1 pascal.hep.osaka Thu Apr 23 17:54 - 17:55 (00:00)
+이름 접속기록
Kazuhiro Hayama -
Tatsusya
Narikawa
(588 분 )
narikawa pts/7 :pts/8:S.0 Wed Jun 3 16:49 - 16:51 (00:01)
narikawa pts/8 pascal.hep.osaka Wed Jun 3 16:49 - 16:51 (00:01)
…
narikawa pts/1 pascal.hep.osaka Thu Apr 23 17:55 - 17:56 (00:00)
narikawa pts/1 pascal.hep.osaka Thu Apr 23 17:54 - 17:55 (00:00)
Hideyuki Tagoshi -
Koh Ueno
(250 분 )
ueno pts/19 :pts/18:S.0 Sat May 2 19:37 - 20:21 (00:43)
ueno pts/18 pascal.hep.osaka Sat May 2 19:37 - 20:21 (00:43)
…
ueno pts/8 pascal.hep.osaka Tue Apr 21 21:00 - 21:07 (00:06)
Hirotaka
Yuzurihara
(28 분 )
yuzu pts/19 :pts/3:S.0 Tue Jun 9 06:09 - 06:09 (00:00)
yuzu pts/3 pascal.hep.osaka Tue Jun 9 06:09 - 06:09 (00:00)
…
yuzu pts/24 pascal.hep.osaka Mon Apr 27 17:39 - 17:48 (00:09)
18
• LDG central monitoring system• Ganglia installed on all KISTI
resources• ce04.sdfarm.kr, a condor server,
gathers ganglia information of LDG WNs.• 8649 port is open to the watchtower.
• 129.89.57.50(watchtower.phys.uwm.edu)
LDG WatchTower
19
• OS: Scientific Linux 6.1 ( ?)• Batch system for managing compute jobs: Condor-7.8.7 ( ?)• LIGO Data Grid : 5.2.2 https://www.lsc-group.phys.uwm.edu/daswg/download/repositories.html• More than 200 packages
Soft Wares Deployed
Packages Description
FrameL for data_frame manipulation MetaIO for LIGO_LW files metadata manipulation LAL Suite Related Project: PYLAL
LIGO Algorithm Library [LAL] + LAL based Applications [LALApps]
GLUE Grid LSC User Environment FrameCPP (deprecated) LDAS-TOOLS
C++ interface to access frame structures
GDS LIGO Global Diagnostics System NDS2-Client Related Project: PYNDS
Part of DMT offline that allow the user to down-load LIGO data from the V2 LIGO Network Data Servers.
Matapps-utilitites A collection of MATLAB® based applications for LIGO data analysis
LVAlert LIGO/Virgo Alert Tools GRACEdb Gravitational Wave Candidate Event Database LARS LIGO Archival Service LIGO-common Simple setup of Python ligo namespace
GSTLAL GSTLAL provides a self-contained suite of GStreamer elements (and dependencies) that
expose gravitational-wave data analysis tools from the LAL library for use in GStreamer signal-processing pipelines.
GST-Plugins A collection of scientific visualization plugins for GStreamer using Cairo-powered graphics
Mathematical operations plugins for GStreamer Low-Latency Ligo low-latency data distribution server initialization
VOEvent VOEvent is the standardized language used to report observation and for describing observations of immediate astronomical events
20
III. Conclusion• We have briefly introduced computing resources, en-
vironments and operation status of the KISTI GSDC LDG T3 center.
• We hope to develop a good collaboration in the com-puting and data management of KAGRA in the future.
21
THANK YOU감사 ( 感謝 ) 합니다