a new building data center upgrade capacity and technology step
DESCRIPTION
A New Building Data Center Upgrade capacity and technology step. 2011 – May t he 4 th – Hepix spring meeting Darmstadt (de) Pascal Trouvé (Facility Manager CC IN2P3). Summary :. Latest improvements of the current computing room New computing room - Project scope - PowerPoint PPT PresentationTRANSCRIPT
A New Building Data Center Upgrade capacity and technology step
2011 – May the 4 th– Hepix spring meeting
Darmstadt (de)
Pascal Trouvé (Facility Manager CC IN2P3)
Summary :
Latest improvements of the current computing room New computing room - Project scope New Building – New architecture Technical commissioning
– Virtual tour– Technical data & figures
On going projects– Technical infrastructures phase II– Electrical power supply challenge
Future development project
[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011
Latest improvements of the computing room
POWER
– New transformers have been installed (01/2010) : 3 X 1 MVA – Diesel generator (1.1 MVA)– Second electrical low voltage master distribution panel installed
(one dedicated for Operation power [racks and cooling], one dedicated to infrastructure power)
– Reactive energy compensation panel ( 4 capacitors banks) - savings– 4 Power distribution panels replaced + Source Transfer Switch for
critical Network racks installed– News batteries farm – sealed lead battery – cell watch system
[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011v
Last improvements of the computing room
COOLING : – A third cooling unit (600 kW) has been installed on roof top,1800 kW
available– Cooling capacity is now maximum complying with environmental
regulation HAZARDS
– News fire safety detection system and monitoring ( SSI/CMSI)– New extinguishers fleet and reviewed lay out
INFRASTRUCTURE – High density confined racks standardization– InRow air-conditioning racks– PDUs suited to equipment specifications, and monitored
[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011
IMPROVMENT INPUTS
POWER No power distribution problems occurred over the last year Better batteries back up capacity coupled with the GE
COOLING Still not fully redundant (short of 1600 kW) on the existing
computing room No cooling issues over the last summer, redundancy will be
reached with the new computing room commissioning HAZARDS
Better fire safety with modern fire detection system Decrease of battery’s Hydrogen Hazard with sealed technology
INFRASTRUCTURE Better global efficiency and sustainability Higher general density
[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011
Power exemplification
6
New aeros on roof top
2 New chillers unit
[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011
Confinement and Water cooled racks
7
confinement
InRaws cooled racks
All the new hardware is installed in “confined racks”, including disks
[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011
Project scope New computing room
The identified needs :~ 60 users groups increased demand Meet LHC’s requirements To meet the planned upgrade of LHC Important identified request for « astroparticules” research activity
[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011
Project scope - Evolution plan – new
The new computing room is an extension of the existing computing room (3 MW power + cooling in 2011)
Target plan:
201150 racks0.6 MW
2015125 racks1.5 MW
2019216 racks3.2 MW
9
Total power 3 MW
Total power 6 MW
Total power 9 MW
[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011
A careful hardware selection
10
IBM iDataplexIntel 5430 LV –
2.66 GHz – 50 W16 GB
DELL 1U 1950b 3 GHz
16 GBBlades DELL
E5540 (2.53 Ghz)48 GB
Dell C6100X5650 – 2.66 GHz
72 GB
[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011
New Building – New architecture
[email protected] Hepix Spring Meeting - Darmstadt - Mai 5 th 2011
31 mars 2011
Chiller units
Cold Water pumps stations
Special construction design – heavy load roof structure
Heavy duty Elevator 3 tons load
12Dominique Boutigny
Chilled water tank 30 m3
Infrastructure design scope
13
Fully confined architecture
All the water pipes, electrical distribution and other services are routed from the ceiling
[email protected] Hepix Spring Meeting - Darmsatdt - Mai 4 th 2011
New computing room
3 levels of 850 Sq. Meters each [ machine room for 850 sq. meters and 2 technical levels including the roof top] + heavy load elevator
We adopted a simplified “Design and build procedure” (single company contracted) delivered the building within18 months
Due to budget constraints the new computer room start with a limited technical infrastructure, but has a modular design that allowed scalable upgrades
First years, room dedicated to worker nodes only
Chilled water and electricity distribution and back up, technical rooms are designed for the 2019 target value
[email protected] Hepix Spring Meeting - Darmstadt - Mai 5 th 2011
The New Data Center Building
[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011
Setting up of a new building live !
• 12 months construction work• >= 630 tons of concrete• >= 70 tons of steel framework•
Building work started end of April 2010
Targeted delivery date targeted February 1st 2011Final delivery date : March 14 th 2011Technical commissioning : undergoingGo live production ~ Mai 24 th 2011
16
Virtual Tour
24 APC racks : InRaw 20 KW eachConfined + workers + 2 Penduit racks with Nexus + switch Cisco
To be completed by end of year :
26 APC racks InRaw 20 KW and fully
confined area
Still 600 sqr meters available ……
[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011
Computing room phase I
17
Technical equipment and installation on roof top
• 2 Chillers units 800 KW each• Chilled water tank 30 tons –
30 min back up• Insulated piping and electrical
tracing• Spare place for future
upgrades
[email protected] Hepix Spring Meeting - Darmstadt - Mai 5 th 2011
18
Power Distribution
• 2 UPS Galaxy 7000 – 600 KVa• Main Switch board 2 MW• Redundant Air cooling unit
[email protected] Hepix Spring Meeting - Darmstdt - Mai 4 th 2011
Fire protection
• “Vesda” permanent detection • Conventional fire detection system level one• Sprinkler (automatic water extinguisher system)
level 2• Monitoring
Power distribution structure
19
Cellules HTA
TGBT UPS critiques +
services
2 500 kVA
TGBT UPS
critiques
400 kVA
TGBT Workers
2 400 kVA
N Modules
clim
Critics
ServicesWorkers600 kW HQ
115 kW HQ
574 kW HQ
N+1
600 kVA
N
600 kVA
N NN
N
1 000 kVA
N+1
1 000 kVA
N+1
800 kVA
N
630 kVA
N
200 kVA
N Modules
clim
N Modules
climCritiques
N x (1+1/5)Modules
climServicesN
N+1
800 kW
N+1
800 kW
[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011
20
Phase II
Additional Equipment & facilities
• Additional power transformer• Third chiller unit 400 KW• Second Batteries farm• Piping and water pumps• Additional power switch boards
[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011
Project launched to be delivered by end of October 2011
A “ready to plug in” for 50 more racks units ……
On going projects
Main Electrical Supply redundancy project
21
2013 – 1,95/ 4 MW 1,95 MW will be backed
up by the power supplier
2016 -2019 - 4 / 9MW 4 MW will be guaranteed
by the power supplier
2011 - 1,7/ 6 MW
Final decision taken to face a new challenge
[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011
In the first phase the electrical redundancy will be minimal
On going projects
Future Development project
22
Amphitheatre accessible to the public
Multimedia center
Outreach center
Dedicated Training center
A “Hub” for fundamental
research, University and
public information
An office space was part of the original extension project but dropped due to budget limitation
Project Re-formatted with new functionalities added
Very well perceived by the Region and the city community (Grand Lyon)
[email protected] Hepix Spring Meeting - Darmstadt - Mai 5 th 2011