1

PDC Site Update at HP-CAST NTIG 1st April 2008 by Peter Graham

PDC Site Update

at HP-CAST NTIGApril 1st 2008

Linköping

byPeter Graham graham@kth.se PDC

Kungl Tekniska Högskolan

2

PDC Site Update at HP-CAST NTIG 1st April 2008 by Peter Graham

PDC premises since 2004

3

PDC Site Update at HP-CAST NTIG 1st April 2008 by Peter Graham

4

PDC Site Update at HP-CAST NTIG 1st April 2008 by Peter Graham

PDC a Centre for HP Scientific Computing (HP=High Performance)

HP Itanium cluster Lucidor

5

PDC Site Update at HP-CAST NTIG 1st April 2008 by Peter Graham

upgraded in 2007 to…

HP Itanium cluster Lucidor 2

6

PDC Site Update at HP-CAST NTIG 1st April 2008 by Peter Graham

Lucidor2 at a glance

• The system contains 106 nodes, each with four Itanium2 (McKinley) 1.3GHz CPUs. 22 nodes have 48 Gb RAM and the rest have 32 Gb RAM. At least 64 nodes available for general (i.e. SNIC) user.

• The interconnect, Myrinet 2000, now uses MX stack which improves latency.

• Myrinet M3-E128 switch is populated with 112 ports. Each card/port has a data rate of 2+2 Gbit/s, all through 50/125 multi-mode fiber.

• Linux distribution CentOS 5.

7

PDC Site Update at HP-CAST NTIG 1st April 2008 by Peter Graham

Latest HP addition ”Key”

HP SMP system (donated to KTH in 2008)

Key is a shared memory system consisting of 32 1.6 GHz cores of IA64 (Intel) type with 18 MB cache. The total main memory will be 256 GB.

(named after Ellen Key)

8

PDC Site Update at HP-CAST NTIG 1st April 2008 by Peter Graham

Current systems

• Lenngren, 442 nodes, Dell 1850• Lucidor2, 106 nodes, HP Itanium2• SweGrid, 100 nodes, South Pole Pentium 4• SBC , 354 nodes, Dell P4 and South Pole Athlon XP• Hebb, IBM BlueGene/L

New systems:• Key , HP SMP, 16 nodes• Ferlin, 680 nodes, Dell M600 blade• SweGrid2, 90 nodes, Dell M600 blade• Climate and turbulence system, under joint

procurement with NSC, for SMHI, MISU at SU and Dep of Mechanics at KTH

9

PDC Site Update at HP-CAST NTIG 1st April 2008 by Peter Graham

Infrastructure power & cooling

• Change of transformer from 800 kVA to 2 MVA, done• Upgrading of UPS from 400 kVA to 1100 kVA, in progress• Diesel generator 400 kVA, existing• Upgrade of cooling exchanger, done• Adding 300 kW APC cooling hut for Ferlin and SweGrid2• Addition of 300 kW chiller for redundant cooling , in progress

10

PDC Site Update at HP-CAST NTIG 1st April 2008 by Peter Graham

Price-performance vs energy

• Power per node 250-400W• Energy cost for 300W over 4 years is nearly 15 kSEK• If you pay 15 kSEK per node you spend equal amount

on investment and energy• To develop more energy efficient nodes will give a

competitive advantage• We would prefer to spend money on application experts

rather than on energy bills

11

PDC Site Update at HP-CAST NTIG 1st April 2008 by Peter Graham

Summing up

• PDC is tripling power capacity to meet the need of the new systems coming in

• High density cooling is required for the new systems with around or above 20 kW per rack

• Energy efficiency, both in regards to cost but also out of environmental concern is becoming more important

• Our new patch cables… -->>(for the UPS)