![Page 1: CERN - IT Department CH-1211 Genève 23 Switzerland t Tier0 database extensions and multi-core/64 bit studies Maria Girone, CERN IT-PSS LCG](https://reader034.vdocuments.site/reader034/viewer/2022051620/56649e925503460f94b97a8f/html5/thumbnails/1.jpg)
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Tier0 database extensions and multi-core/64 bit studies
Maria Girone, CERN IT-PSS
LCG 3D Database Workshop, 12-13 June 2007
![Page 2: CERN - IT Department CH-1211 Genève 23 Switzerland t Tier0 database extensions and multi-core/64 bit studies Maria Girone, CERN IT-PSS LCG](https://reader034.vdocuments.site/reader034/viewer/2022051620/56649e925503460f94b97a8f/html5/thumbnails/2.jpg)
CERN - IT DepartmentCH-1211 Genève 23
[email protected] PSS group meeting, May 2007 - 2
Outline
• Current set-up
• Quad-core/64 bit tests
• Resource allocation for 2008
![Page 3: CERN - IT Department CH-1211 Genève 23 Switzerland t Tier0 database extensions and multi-core/64 bit studies Maria Girone, CERN IT-PSS LCG](https://reader034.vdocuments.site/reader034/viewer/2022051620/56649e925503460f94b97a8f/html5/thumbnails/3.jpg)
CERN - IT DepartmentCH-1211 Genève 23
[email protected] PSS group meeting, May 2007 - 3
Current set-up at Tier0
• RAC on commodity hardware– Full redundancy– Linux RHES4 32bit as OS platform– Oracle ASM as volume Manager– 10g version of Oracle (10.2.0.3)– Dual-CPU P4 Xeon servers with 4GB of DDR2 400
memory each
Disk array
Disk array
FC switch
FC switch
Ethernet switch
![Page 4: CERN - IT Department CH-1211 Genève 23 Switzerland t Tier0 database extensions and multi-core/64 bit studies Maria Girone, CERN IT-PSS LCG](https://reader034.vdocuments.site/reader034/viewer/2022051620/56649e925503460f94b97a8f/html5/thumbnails/4.jpg)
CERN - IT DepartmentCH-1211 Genève 23
[email protected] PSS group meeting, May 2007 - 4
Main Operational Aspects
Service Size
– 110 mid-range servers and 110 disk arrays (~1100 disks)
– In other words: 220 CPUs, 440GB of RAM, 300 TB of raw disk space
• Several production clusters– One production cluster per LHC experiment for offline
applications, varying from 4 to 8-node clusters – Online test Atlas cluster– COMPASS cluster
• Several validation and test clusters– 1 or 2 per LHC experiment of 2-nodes – Some hardware allocated for internal use/tests
• Service responsibilities– 6 DBAs in the team – 24x7 service on best effort for the production service– 8x5 coverage for streams
![Page 5: CERN - IT Department CH-1211 Genève 23 Switzerland t Tier0 database extensions and multi-core/64 bit studies Maria Girone, CERN IT-PSS LCG](https://reader034.vdocuments.site/reader034/viewer/2022051620/56649e925503460f94b97a8f/html5/thumbnails/5.jpg)
CERN - IT DepartmentCH-1211 Genève 23
[email protected] PSS group meeting, May 2007 - 5
Current set-up
![Page 6: CERN - IT Department CH-1211 Genève 23 Switzerland t Tier0 database extensions and multi-core/64 bit studies Maria Girone, CERN IT-PSS LCG](https://reader034.vdocuments.site/reader034/viewer/2022051620/56649e925503460f94b97a8f/html5/thumbnails/6.jpg)
CERN - IT DepartmentCH-1211 Genève 23
[email protected] PSS group meeting, May 2007 - 6
2007 hardware allocation
2007 Experiment/
activity Production Integration/
Pilot/Test On-line test
ALICE On PDB - 1 x 2 nodes (PVSS tests)
ATLAS 1 x 6 nodes 2 x 2 nodes 1 x 6 nodes
CMS 1 x 8 nodes 2 x 2 nodes
LHCB 1 x 6 nodes 1 x 2 nodes
WLCG 1 x 8 nodes 2 x 2 nodes -
PDB (alice, harp, totem)
1 x 4 nodes - -
COMPASS 1 x 4 nodes - -
3D
3 nodes (downstream capture for atlas and
lhcb) 1 node (OEM)
1 x 2 nodes -
Test
1 x 4 nodes (coral) 1 x 6 nodes (cool)
1 x 2 nodes (recovery)
1 x 6 nodes (PVSS IT-CO)
![Page 7: CERN - IT Department CH-1211 Genève 23 Switzerland t Tier0 database extensions and multi-core/64 bit studies Maria Girone, CERN IT-PSS LCG](https://reader034.vdocuments.site/reader034/viewer/2022051620/56649e925503460f94b97a8f/html5/thumbnails/7.jpg)
CERN - IT DepartmentCH-1211 Genève 23
[email protected] PSS group meeting, May 2007 - 7
Quad-core/64 bit tests
• We have tested a 2x Intel quad-core CPUs and 16GB of RAM and compared with current Tier0 hardware (2x P4 CPUs)
• Natural point to introduce 64-bit Oracle servers
• All results are available at https://twiki.cern.ch/twiki/bin/view/PSSGroup/QuadCoreTests
![Page 8: CERN - IT Department CH-1211 Genève 23 Switzerland t Tier0 database extensions and multi-core/64 bit studies Maria Girone, CERN IT-PSS LCG](https://reader034.vdocuments.site/reader034/viewer/2022051620/56649e925503460f94b97a8f/html5/thumbnails/8.jpg)
CERN - IT DepartmentCH-1211 Genève 23
[email protected] PSS group meeting, May 2007 - 8
Response time measurements
• The quad-core server can sustain the same logical IO workload as 5 of the servers currently deployed in production
O r a c l e L o g i c a l I O T e s t ( J L O C I ) - E x e c u t i o n t i m e v s . c o n c u r r e n c y
0
5 0
1 0 0
1 5 0
2 0 0
2 5 0
3 0 0
3 5 0
4 0 0
4 5 0
1 2 3 4 5 6 8 1 0 1 2 1 4 1 6 1 8 2 0 2 4 2 8 3 2
T h r e a d s ( p )
Execution time, R(p)
2 C P U - 1 i n s t R A C ( 6 N * 2 C P U ) 4 C O R E R A C ( 4 N * 2 C P U )
• Measurements of CPU to memory access were performed using a ORACLE workload (Jonathan Lewis Oracle Computing Index query JLOCI)
![Page 9: CERN - IT Department CH-1211 Genève 23 Switzerland t Tier0 database extensions and multi-core/64 bit studies Maria Girone, CERN IT-PSS LCG](https://reader034.vdocuments.site/reader034/viewer/2022051620/56649e925503460f94b97a8f/html5/thumbnails/9.jpg)
CERN - IT DepartmentCH-1211 Genève 23
[email protected] PSS group meeting, May 2007 - 9
Performance testing with PhEDEx
A single quad core server is able to handle PhEDEx-like workload (a transaction oriented application) even more efficiently then a 6-node RAC
6-node RAC
Quad-core server
![Page 10: CERN - IT Department CH-1211 Genève 23 Switzerland t Tier0 database extensions and multi-core/64 bit studies Maria Girone, CERN IT-PSS LCG](https://reader034.vdocuments.site/reader034/viewer/2022051620/56649e925503460f94b97a8f/html5/thumbnails/10.jpg)
CERN - IT DepartmentCH-1211 Genève 23
[email protected] PSS group meeting, May 2007 - 10
Quad-core performance and Oracle Streams
The source DB on a 2-node RAC on P4, destination (apply) machine on a single-core and quad-core DB server
The quad-core results show a 70% increase in performance compared to the current production HW
![Page 11: CERN - IT Department CH-1211 Genève 23 Switzerland t Tier0 database extensions and multi-core/64 bit studies Maria Girone, CERN IT-PSS LCG](https://reader034.vdocuments.site/reader034/viewer/2022051620/56649e925503460f94b97a8f/html5/thumbnails/11.jpg)
CERN - IT DepartmentCH-1211 Genève 23
[email protected] PSS group meeting, May 2007 - 11
Quad core/64 bit tests (cont’d)• Measured power consumption per core
showed a gain in efficiency for quad-core compared to current RAC production of a factor 2
• Installation procedures for quad-core and RHEL 4 do not need additional effort compared to the existing (there are minor differences with current production RACs)
• SAN configuration and I/O throughput have been tested and are unchanged from the existing RAC configuration.
![Page 12: CERN - IT Department CH-1211 Genève 23 Switzerland t Tier0 database extensions and multi-core/64 bit studies Maria Girone, CERN IT-PSS LCG](https://reader034.vdocuments.site/reader034/viewer/2022051620/56649e925503460f94b97a8f/html5/thumbnails/12.jpg)
CERN - IT DepartmentCH-1211 Genève 23
[email protected] PSS group meeting, May 2007 - 12
Hardware allocation in 2008
• Production databases for LHC:– 3-4 node clusters built with quadcore CPU
machines (24-32 cores per cluster)– 48-64 GB of RAM per cluster– Planning for >10k IOPS – TBs of mirrored space
• Integration and test systems:– Old single core CPU hardware– Usually 2 nodes per cluster– Usually 24-32 disks
• 64bit version of Linux and Oracle software
![Page 13: CERN - IT Department CH-1211 Genève 23 Switzerland t Tier0 database extensions and multi-core/64 bit studies Maria Girone, CERN IT-PSS LCG](https://reader034.vdocuments.site/reader034/viewer/2022051620/56649e925503460f94b97a8f/html5/thumbnails/13.jpg)
CERN - IT DepartmentCH-1211 Genève 23
[email protected] PSS group meeting, May 2007 - 13
Conclusions
• Tests on dual quad-core Xeon processors look very promising and were compared with dual-CPU P4 http://twiki.cern.ch/twiki/bin/view/PSSgroup/QuadCoreTests
• We have decided to base our next order on this choice
• Natural move to Oracle 64 bit
• Migration tools are being prepared and tested to minimize the downtime of the production RACs