GPU Cluster for Scientific Computing

Zhe Fan, Feng Qiu, Arie Kaufman, Suzanne Yoakum-StoverCenter for Visual Computing and Department of Computer Science, Stony Brook University

http://www.cs.sunysb.edu/~vislab/projects/gpgpu/GPU_Cluster/GPU_Cluster.html

Stony Brook Visual Computing ClusterStony Brook Visual Computing Cluster

• GPU ClusterGPU Cluster

• 35 nodes with nVIDIA GeForce FX 5800 Ultra35 nodes with nVIDIA GeForce FX 5800 Ultra

• Gigabit EthernetGigabit Ethernet

• 70 Pentium Xeon 2.4GHz CPUs70 Pentium Xeon 2.4GHz CPUs

• 35 VolumePro 100035 VolumePro 1000

• 9 HP Sepia-2A with ServerNet II9 HP Sepia-2A with ServerNet II

Stony Brook Visual Computing ClusterStony Brook Visual Computing Cluster

• GPU ClusterGPU Cluster

• 35 nodes with nVIDIA GeForce FX 5800 Ultra35 nodes with nVIDIA GeForce FX 5800 Ultra

• Gigabit EthernetGigabit Ethernet

• 70 Pentium Xeon 2.4GHz CPUs70 Pentium Xeon 2.4GHz CPUs

• 35 VolumePro 100035 VolumePro 1000

• 9 HP Sepia-2A with ServerNet II9 HP Sepia-2A with ServerNet II

LBM on the GPULBM on the GPUApplication: large-scale CFD simulations using LatticeApplication: large-scale CFD simulations using Lattice

Boltzmann Model (LBM)Boltzmann Model (LBM)

LBM Computation:LBM Computation:

• Particles stream along lattice linksParticles stream along lattice links

• Particles collide when they meet at a siteParticles collide when they meet at a site

Map to GPU:Map to GPU:

• Pack 3D lattice states into a series of 2D texturesPack 3D lattice states into a series of 2D textures

• Update the lattice with fragment programsUpdate the lattice with fragment programs

LBM on the GPULBM on the GPUApplication: large-scale CFD simulations using LatticeApplication: large-scale CFD simulations using Lattice

Boltzmann Model (LBM)Boltzmann Model (LBM)

LBM Computation:LBM Computation:

• Particles stream along lattice linksParticles stream along lattice links

• Particles collide when they meet at a siteParticles collide when they meet at a site

Map to GPU:Map to GPU:

• Pack 3D lattice states into a series of 2D texturesPack 3D lattice states into a series of 2D textures

• Update the lattice with fragment programsUpdate the lattice with fragment programs

Scale up LBM to the GPU ClusterScale up LBM to the GPU Cluster• Each GPU computes a sub-latticeEach GPU computes a sub-lattice

• Particles stream out of the sub-latticeParticles stream out of the sub-lattice

1.1. Gather particle distributions in a textureGather particle distributions in a texture

2.2. Read out from GPU in a single operationRead out from GPU in a single operation

3.3. Transfer through GigaE (MPI)Transfer through GigaE (MPI)

4.4. Write into neighboring GPU nodesWrite into neighboring GPU nodes

• Network performance optimization:Network performance optimization:

1.1. Conduct network transfer while computingConduct network transfer while computing

2.2. Schedule to reduce the likelihood of interruptionSchedule to reduce the likelihood of interruption

3.3. Simplify the connection patternSimplify the connection pattern

Scale up LBM to the GPU ClusterScale up LBM to the GPU Cluster• Each GPU computes a sub-latticeEach GPU computes a sub-lattice

• Particles stream out of the sub-latticeParticles stream out of the sub-lattice

1.1. Gather particle distributions in a textureGather particle distributions in a texture

2.2. Read out from GPU in a single operationRead out from GPU in a single operation

3.3. Transfer through GigaE (MPI)Transfer through GigaE (MPI)

4.4. Write into neighboring GPU nodesWrite into neighboring GPU nodes

• Network performance optimization:Network performance optimization:

1.1. Conduct network transfer while computingConduct network transfer while computing

2.2. Schedule to reduce the likelihood of interruptionSchedule to reduce the likelihood of interruption

3.3. Simplify the connection patternSimplify the connection pattern

Times Square Area of NYCTimes Square Area of NYC

Flow StreamlinesFlow Streamlines

• 0.31 second / step on 30 GPUs0.31 second / step on 30 GPUs

• 4.6 times faster than software version on 30 CPUs4.6 times faster than software version on 30 CPUs

Times Square Area of NYCTimes Square Area of NYC

Flow StreamlinesFlow Streamlines

• 0.31 second / step on 30 GPUs0.31 second / step on 30 GPUs

• 4.6 times faster than software version on 30 CPUs4.6 times faster than software version on 30 CPUs

AcknowledgementsAcknowledgements

• NSF CCR0306438NSF CCR0306438

• Department of Department of Homeland Security, Homeland Security, Environment Environment Measurement LabMeasurement Lab

• HP HP

• TerareconTerarecon

AcknowledgementsAcknowledgements

• NSF CCR0306438NSF CCR0306438

• Department of Department of Homeland Security, Homeland Security, Environment Environment Measurement LabMeasurement Lab

• HP HP

• TerareconTerarecon

GPU Cluster / CPU Cluster SpeedupGPU Cluster / CPU Cluster Speedup

• Each node computes an 80 x 80 x 80 sub-latticeEach node computes an 80 x 80 x 80 sub-lattice

• GeForce FX 5800 Ultra / Pentium Xeon 2.4GHzGeForce FX 5800 Ultra / Pentium Xeon 2.4GHz

GPU Cluster / CPU Cluster SpeedupGPU Cluster / CPU Cluster Speedup

• Each node computes an 80 x 80 x 80 sub-latticeEach node computes an 80 x 80 x 80 sub-lattice

• GeForce FX 5800 Ultra / Pentium Xeon 2.4GHzGeForce FX 5800 Ultra / Pentium Xeon 2.4GHzDispersion PlumeDispersion PlumeDispersion PlumeDispersion Plume

• 1.66 km x 1.13 km1.66 km x 1.13 km

• 91 blocks 91 blocks

• 851 buildings851 buildings

• 480 x 400 x 80 lattice480 x 400 x 80 lattice

and Large-Scale Simulation