ivec.org team quokka: australia’s first foray into the student cluster challenge rebecca...
TRANSCRIPT
ivec.org
Team Quokka: Australia’s First Foray into the Student
Cluster Challenge
Rebecca Hartman-Baker, Ph.D.Senior Supercomputing Specialist & SCC Team Coach
Happy Cheeks, Ph.D.Supercomputing Quokka & Team Mascot
http://www.facebook.com/happycheeksquokka
ivec.org
ivec.org
Outline
I. Student Cluster Competition
II. Building & Training SCC Team
III. Cluster Design
IV. Science Applications
V. Competition
ivec.org
ivec.org
I. STUDENT CLUSTER COMPETITIONiVEC Student Cluster Competition Training Team, 2013
Drag picture to placeholder or click icon to add
ivec.org
Student Cluster Competition: Introduction
• 48-hour non-stop computing showdown held at annual supercomputing conference• Held in Denver, Colorado, USA in 2013
• Teams of undergraduates design, build, run apps on cluster• Power constraint ~3000W
(standard track)• $2500 equipment constraint
(commodity track)
ivec.org
SCC: Introduction• Key cluster architecture rules:• Machine must contain only publicly available
components• All components must be turned on at all times
(low-watt idle okay)• When running applications, must not exceed 26 A
@ 120 V power draw (~3120 W)
ivec.org
SCC: Introduction• Applications: HPCC + 3 known apps, 1
mystery app• 2013 apps: Graphlab, WRF, Nemo5, + Flying
Snakes (OpenFOAM)
• Teams judged on throughput of work provided by judges, plus interview to determine depth of understanding
ivec.org
SCC: History• First held at 2007 Supercomputing
conference, SC07• Brainchild of Brent Gorda, now
General Manager, Intel High-Performance Data Division (formerly Whamcloud)
• Now three Student ClusterCompetitions/year:• China (April)• ISC (June)• SC (November)
ivec.org
SCC: iVEC’s Motivation
• Increase computational science literacy in WA
• Develop future users/employees• Train professional workforce for local industry• Exposure for iVEC in international HPC
community• It sounded like fun!
ivec.org
Starting iVEC’s SCC Team• Began by raising interest at iVEC partner
universities• Contacted iVEC directors at universities, got leads
for whom to contact• First interest came from ECU (3 students)• Other unis followed
ivec.org
Sponsorship for SCC Team• SGI
• NVIDIA
• Allinea
• Rio Tinto
ivec.org
Sponsorship for SCC Team• Discussed hardware sponsorship with Cray &
SGI• SGI first to commit, hardware + travel money
• Solicited financial sponsorship from mining companies in WA• Rio Tinto committed to sponsor 3 students
• Obtained software & hardware sponsorship from Allinea & NVIDIA
ivec.org
Team Hardware Sponsorship
• Most important sponsorship to get• SGI very enthusiastic about sponsoring team• Put best person in Asia-Pacific region on
project• Todd helped team: • Select machine architecture• Determine software stack• Set the machine up in Perth & at competition
ivec.org
Team Hardware Sponsorship
• When team decided to use GPUs in cluster, NVIDIA loaned us 8 K20X GPUs
• Received free of charge through academic program (had to return after competition )
ivec.org
Team Travel Sponsorship• Travel to competition very expensive• Budget: $3000/student• SGI committed enough for half of team• Solicited support from mining companies in
WA, successful with Rio Tinto
ivec.org
Team Software Sponsorship
• I “won” license for Allinea software• I asked Allinea to sponsor license for team
instead• Allinea provided license for MAP & DDT
products• MAP: simple profiling tool, very useful to novice
users• DDT: parallel debugger, intuitive GUI
ivec.org
Team Composition• Breakdown:• 3 Computer Science/Games majors from ECU• 2 Physics/Computer Engineering majors from
UWA• 1 Geophysics major from Curtin
• Each student assigned areas of expertise (1 primary, 2 secondary)
• At beginning of training, I facilitated students’ development of team norms (standards of behavior) that proved very effective• No conflicts, no inappropriate behavior
ivec.org
Cluster Design• Designing cluster, generally the following
must be considered:• Cost• Space• Utility• Performance• Power Consumption• Cost
ivec.org
Cluster Design• Designing cluster, generally the following
must be considered:• Cost• Space• Utility• Performance• Power Consumption• Cost
ivec.org
Cluster Design• Architecture choices:• All CPU nodes• All accelerator nodes• Hybrid CPU/accelerator
• Accelerator choices:• NVIDIA Tesla• Intel Xeon Phi• Combination (?)
ivec.org
Cluster Architecture• 2 Pyramid nodes• 4 x NVIDIA K20X• 2 x Intel Ivy Bridge 12-
core 64 GB
• 8 Hollister nodes• 2 x Intel Ivy Bridge 12-
core 64 GB
• Infiniband interconnect
• Stay within power budget by running only GPUs or CPUs
ivec.org
Cluster Architecture• Chose CPU/GPU hybrid architecture• For good LINPACK performance• Potential accelerated mystery app• Maximize flops per watt performance
ivec.org
Cluster Software• CentOS 6• Ceph filesystem• Open-source Software• GCC• OpenMPI• Numerical libraries
• FFTW• PETSc• NetCDF• HDF5
• Proprietary software• Intel compiler/MKL• Allinea DDT/MAP• PGI compiler• CUDA
ivec.org
Cluster Software: Ceph• Each node has > 1TB disk, need parallel
filesystem• Could use Lustre, however issues with losing
data if one node fails• Ceph: distributed object store and file system
designed to provide excellent performance, reliability and scalability
ivec.org
Cluster Software: Ceph• Ceph object-storage system with traditional
file-system interface with POSIX semantics• Looks like regular filesystem• Directly mounted in recent CentOS kernel
• Underneath, Ceph keeps several copies of files balanced across hosts• Metadata server cluster can expand/contract to fit
file system• Rebalance dynamically to distribute data (weighted
distribution if disks differ in size)
ivec.org
Applications• High-Performance LINPACK• Graphlab• NEMO5• WRF• Mystery Application – Flying Snakes!
ivec.org
HIGH-PERFORMANCE LINPACK
ivec.org
Linpack History• Linear algebra library written in Fortran• Benchmarking added in late 1980s to
estimate calculation times• Initial releases used fixed matrix sizes 100
and 1000• Arbitrary problem size support added in 1991• LAPACK replaced the Linpack library for
linear algebra, however Linpack benchmarking tool still used today
ivec.org
HPL Standard• Released in 2000 re-written in C and
optimized for parallel computing• Uses MPI and BLAS
• The standard benchmark used measure supercomputer performance• Used to determine the Top500• Also used for stress testing and maintenance
analysis
ivec.org
CUDA HPL• CUDA-accelerated Linpack released by
NVIDIA available on developer zone• Uses the GPU instead of the CPU and limited
to GPU memory• Popularity gaining with GPU providing better
flops/watt• Standard for HPL runs in Student Cluster
Competitions
ivec.org
2006 2007 2008 2009 2010 2011 2012 2013 20140.1
1
10
Student Cluster Competition Linpack Scores
TF
Year
ivec.org
GRAPHLAB
ivec.org
GraphLab• Toolkit for graph algorithms• Topic Modelling• Graph Analytics• Clustering• Collaborative Filtering• Graphical Models• Computer Vision
ivec.org
Graphlab Applications• Page rank (e.g., Google)• Image reconstruction• Recommendation predictions
(e.g., Netflix)• Image stitching (e.g., panoramic photos)
ivec.org
NEMO5
ivec.org
NEMO5• Stands for NanoElectronics MOdeling
Tools • Free for academic use, not exactly open
source• Evolved to current form over 15 years• Developed by Purdue University
ivec.org
NEMO5• NEMO5 designed to model at the
atomic scale• Simulation of nanostructure
properties: strain relaxation, phonon modes, electronic structure, self-consistent Schrodinger-Poisson calculations, and quantum transport
• E.g., modelling Quantum Dots
ivec.org
WRF
ivec.org
WRF• Next-generation mesoscale numerical
weather prediction system• Used for both weather prediction and
research forecasting, throughout the world
ivec.org
MYSTERY APPLICATION: FLYING SNAKES!
ivec.org
Mystery Application• Unknown application, presented at
competition• To prepare, compiled and ran one new code
each week during 2nd semester• Gained experience with different types of
compiles (e.g., edit makefiles, make.inc, cmake, autoconf, etc.)
• Gained familiarity with common errors encountered while compiling, and how to fix them
ivec.org
Flying Snakes!• Aerodynamics of flying snakes• Flying snakes inhabit rainforest canopy in East
Asia & jump between tree branches, gliding to next branch
• Case of fluid dynamics: behavior of air as snake passes through, development of vortices, eddies, etc.
• Modeled with OpenFOAM, open-source computational fluid dynamics toolbox
ivec.org
Competition• Arrived in Denver Thursday before
competition, to acclimate to 15-hour time difference• Visited National Renewable Energy Laboratory to
see supercomputers
• Began setting up on Saturday before competition
• Competition time: Monday evening – Wednesday evening
• Wednesday evening: party at Casa Bonita• Thursday: Pros vs. amateurs competition• Friday: back home
ivec.org
Scenes from the Trip
ivec.org
Scenes from the Trip
ivec.org
Team Booth
ivec.org
Casa Bonita
ivec.org
Taking Down the Booth
ivec.org
Results• Official champion: University of Texas (last
year’s champions too)• Other rankings not given, but we were middle
of pack• Entire team (including coach) learned a lot!• Students have potential leads for jobs &
further study• Plans to coach another team for 2014
ivec.org
Bibliography• CentOS, http://www.centos.org• Ceph,
http://ceph.com/ceph-storage/file-system/• High-Performance Linpack & HPCC,
http://icl.cs.utk.edu/hpcc/• Graphlab, http://graphlab.org• NEMO5,
https://engineering.purdue.edu/gekcogrp/software-projects/nemo5/
• WRF, http://www.wrf-model.org/index.php• Krishnan et al., Lift and wakes of flying snakes,
http://arxiv.org/pdf/1309.2969v1.pdf• OpenFOAM, http://www.openfoam.com
ivec.org
For More Information• iVEC, http://www.ivec.org• Student Cluster Competition,
http://www.studentclustercomp.com
Email: [email protected]