Visualizing Science at Scale with Longhorn

Kelly Gaither

Director of Data & Information Analysis/Research Scientist

Texas Advanced Computing Center

November 16, 2010

Visualization Group at TACC

• 13 Full Time Staff– 6 Ph.D. – 4 Masters– 3 Bachelors

• 7 Students– 2 Undergraduate Students– 4 Masters Students– 1 PhD Student

Visualization Group at TACC

• Areas of Expertise– Remote & Collaborative Visualization– Large Data Visualization– Large Scale GPU Clusters– Large Scale Tiled Displays – Data Mining & Feature Detection

Motivation for Discussing Visualization Issues at Large-Scale

Full-Scale Hurricane Prediction/Recovery

• Where will the next hurricane hit?• What is the projected loss?

Cosmology• How did the earliest

galaxies form?• What was the first star

and when did it form?

Biological Systems• What are the

processes at the sub-cellular level?

• How can we understand chains of reactions within living cells?

Not Just Simulation Any More… Vastly more powerful

instruments and computers have led to an explosion of new data.

Modern science and engineering therefore is about managing and analyzing this data as well as modeling and simulation.

What Role Does Visualization Play in Large-Scale Science

Why Are Pictures So Powerful?How Critical is Visualization to Science?

LonghornFirst NSF XD Visualization Resource

• 256 Dell Dual Socket, Quad Core Intel Nehalem Nodes– 240 with 48 GB shared memory/node (6 GB/core)– 16 with 144 GB shared memory/node (18 GB/core)– 73 GB Local Disk– 2 Nvidia GPUs/Node (FX 5800 – 4GB RAM)

• ~14.5 TB aggregate memory• QDR InfiniBand Interconnect• Direct Connection to Ranger’s Lustre Parallel File

System• 10G Connection to 210 TB Local Lustre Parallel File

System• Jobs launched through SGE

256 Nodes, 2048 Cores, 512 GPUs, 14.5 TB Memory

Kelly Gaither (PI), Valerio Pascucci, Chuck Hansen, David Ebert, John Clyne (Co-PI), Hank Childs

Longhorn Usage Modalities:

• Remote/Interactive Visualization– Highest priority jobs – Remote/Interactive capabilities facilitated through VNC– Run on 3 hour queue limit boundary

• GPGPU jobs– Run on a lower priority than the remote/interactive jobs– Run on a 12 hour queue limit boundary

• CPU jobs with higher memory requirements– Run on lowest priority when neither remote/interactive nor GPGPU

jobs are waiting in the queue– Run on a 12 hour queue limit boundary

Longhorn Queue Structure

qsub -q normal -P vis

Longhorn’s Lustre File System ($SCRATCH)

• OSS’s on Longhorn are built on Dell Nehalem Servers Connected to MD10000 Storage Vaults

• 15 Drives Total Configured into 2 Raid5 pairs with a Wandering Spare• Peak Throughput Speed of the File System is 5.86 GB/sec• Peak Aggregate Speed of the File System is 5.43 GB/sec

Software Available on Longhorn

• Programming APIs: OpenGL, vtk– OpenGL – low level primitives, useful for programming at a

relatively low level with respect to graphics– VTK (Visualization Toolkit) – open source software system for 3D

computer graphics, image processing, and visualization– IDL

• Visualization Turnkey Systems – VisIt – free parallel visualization and graphical analysis tool– ParaView (Parallel Visualization Application) – open source

general purpose visualization system– EnSight – commercial turnkey visualization package target at CFD

visualization– Amira – commercial turnkey visualization package targeted at

visualizing scanned medical data (CAT scan, MRI, etc..)

Connecting to Longhorn/Spur Using VNC

longhornlonghorn

laptopor

workstation

laptopor

workstation

ssh <user>@longhorn.tacc.utexas.eduqsub /share/sge/default/pe_scripts/job.vnctouch ~/vncserver.outtail –f ~/vncserver.out

longhornlonghorn

laptopor

workstation

laptopor

workstation

ssh –L <port>:longhorn.tacc.utexas.edu:<port> <user>@longhorn.tacc.utexas.edu

VNC server on vis node

ivis[1-7|big]

VNC server on vis node

ivis[1-7|big]

longhornlonghorn

laptopor

workstation

laptopor

workstation

vncviewer localhost::<port> automaticport forwarding

to vis node

establishessecure tunnelto longhorn vnc port

localhost connection forwarded to longhorn via ssh tunnel

contains vnc portinfo after job launches

Longhorn Visualization Portalportal.longhorn.tacc.utexas.edu

>3000 jobs submitted through the portal

EnVisionGreg Johnson, Brandt Westing

• Web-based visualization software that allows researchers to develop interactive visualizations intuitively.

• Currently integrated into the Longhorn Visualization Portal but can run independently.

• Began collaborations with ParaView team.

Usage on Longhorn as of October 31 2010

• 498 active projects • 41,629 jobs run on the system• 3,980,297 SUs expended on the system

Visualizing Oil SpillAdam Kubach, Karla Vega, Clint Dawson

• Visualization focused on the overlay of particle movement and satellite or aerial imaging data.

• The particles in the visualization represent the oil spill and their position is either hypothetical or reflect the position of the oil on the surface.

• The data has been visualized using Longhorn and MINERVA, which is an open source geospatial software. The data is generated daily and is approximately 100 GB in size.

H1N1 Flu Outbreak Simulation Greg Johnson, Brandt Westing, TACC; Ned Dimitrov, Lauren Meyers, UT Comp. Bio

• Visualization of a swine flu epidemic spreading throughout North America.

• Epidemic begins in Mexico City. • Visualization classifies

individuals into three groups: susceptible (blue), infected (red), and recovered (green). Available antivirals are shown in purple.

• Cities and transportation links are highlighted in red to indicate large numbers of infected individuals and infectious travelers.

Visualization of Hurricane Ike Greg Johnson, Romy Schneider, John Cazes, Karl Schulz, Bill Barth, TACC; Frank

Marks, NOAA; Fuqing Zheng, University of Pennsylvania; Yonghui Weng, Texas A&M.

• Throughout the 2008 hurricane season, the TACC was an active participant in a NOAA research effort to develop next-generation hurricane models.

• Using up to 40,000 processing cores at once, researchers simulated both global and regional weather models and received on-demand access to Ranger.

• Visualization of Hurricane Ike shows the storm developing in the gulf and making landfall on the Texas coast.

Visualization of Large Scale Turbulent FlowKelly Gaither, Hank Childs, Greg Johnson, Karl Schulz, Cyrus Harrison,

Diego Donzis, Texas A&M; P.K. Yeung, Georgia Tech

• Remote interactive visualization of 17 time-steps of the largest turbulent flow simulation computed to date.

• First time this had been visualized interactively at this scale (40963).

Thank You

Kelly Gaitherkelly@tacc.utexas.edu