1

Hybrid Parallelism for Volume Renderingon Large, Multi- and Many-core Systems

Mark Howison, E. Wes Bethel, Member, IEEE, and Hank Childs

(Invited Paper)

F

Abstract—With the computing industry trending towards multi- andmany-core processors, we study how a standard visualization algorithm,ray-casting volume rendering, can benefit from a hybrid parallelismapproach. Hybrid parallelism provides the best of both worlds: usingdistributed-memory parallelism across a large numbers of nodes in-creases available FLOPs and memory, while exploiting shared-memoryparallelism among the cores within each node ensures that each nodeperforms its portion of the larger calculation as efficiently as possible.We demonstrate results from weak and strong scaling studies, at levelsof concurrency ranging up to 216,000, and with datasets as large as12.2 trillion cells. The greatest benefit from hybrid parallelism lies inthe communication portion of the algorithm, the dominant cost at higherlevels of concurrency. We show that reducing the number of participantswith a hybrid approach significantly improves performance.

Index Terms—Volume visualization, parallel processing

1 INTRODUCTION

Many in the HPC community have expressed concernthat parallel programming languages, models, and ex-ecution frameworks that have worked well to-date onsingle-core massively parallel systems may “face dimin-ishing returns” as the number of computing cores ona chip increase [1]. In this context, we explore the per-formance and scalability of a common visualization al-gorithm – ray-casting volume rendering – implementedwith different parallel programming models and runon both a large supercomputer comprised of six-coreCPUs and a cluster with many-core GPUs. We com-pare a traditional distributed-memory implementationbased solely on message-passing against a “hybrid”implementation, which uses a blend of message-passing(inter-chip) and shared-memory (intra-CPU/intra-GPU)parallelism. The thesis we wish to test is that there areopportunities in the hybrid-memory implementation forperformance and scalability gains that result from usingshared-memory parallelism among cores within a chip.

• M. Howison is with the Center for Computation and Visualization,Brown University, 94 Waterman Street, Providence, RI 02912, USA,and the Computational Research Division, Lawrence Berkeley NationalLaboratory, One Cyclotron Road, Berkeley, CA 94720, USA. Email:mhowison@brown.edu

• E.W. Bethel and H. Childs are with the Computational Research Division,Lawrence Berkeley National Laboratory, One Cyclotron Road, Berkeley,CA 94720, USA Email: {mhowison,ewbethel,hchilds}@lbl.gov

Fig. 1. This 46082 image of a combustion simulation resultwas rendered by our MPI+pthreads implementation run-ning on 216,000 cores of the JaguarPF supercomputer.

Over the years, there has been a consistent and well-documented concern that the overall runtime of large-data visualization algorithms is dominated by I/O costs(e.g., [2]–[4]). During our experiments, we observed re-sults consistent with previous work: there is a significantcost associated with scientific data I/O. In this study,however, we focus exclusively on the performance andscalability of the ray-casting volume rendering algo-rithm, not on parallel I/O performance. This approach isvalid for many visualization use cases, such as creatingmultiple images from a single dataset that fits entirelywithin the memory footprint of a large system, or creat-ing one or more images of data that is already residentin memory, as in the case of in-situ visualization.

Our findings (Section 5) show that there is indeedopportunity for performance gains when using hybrid-parallelism for raycasting volume rendering across a

2

wide range of concurrency levels. The hybrid-memoryimplementation runs faster, requires less memory and,for this particular algorithm and set of implementationchoices (Section 3), requires less communication band-width than the distributed-memory implementation.

This paper is an extension of our previous work [5],and we have added the following new components: (1) aweak scaling study to better understand performance aswe increase problem size along with processor count; (2)a new platform and implementation – a hybrid/CUDAimplementation run on a distributed memory GPU clus-ter – along with both strong and weak scaling studieson that platform to better understand how the changein ratio of cores to nodes impacts performance and scal-ability; (3) a study to determine the optimal image tilesize and shape, based upon the principles of autotuning,for both the CPU and GPU platforms. For the sake ofcompleteness, we reuse many of the figures and conceptsfrom our earlier work.

2 BACKGROUND AND PREVIOUS WORK

2.1 Parallel Volume RenderingVolume rendering is a common technique for display-ing 2D projections of 3D sampled data [6], [7] and iscomputationally, memory, and data I/O intensive. In thequest towards interactivity, as well as to address thechallenges posed by growing data size and complexity,there has been a great deal of work over the years in thespace of parallel volume visualization (see Kaufman andMueller [8] for an overview). The focus of our work hereis on a hybrid-memory implementation (Section 2.2) atextreme concurrencies to take advantage of multi- andmany-core processor architectures.

Our hybrid-memory implementation makes use of adesign pattern common to many parallel volume ren-dering applications that use a mixture of both object-and pixel-level parallelism [9]–[12]. The design employsan object-order partitioning to distribute source datablocks to processors where they are rendered using raycasting [6], [7], [13], [14]. Within a processor, we thenuse an image-space decomposition, similar to Nieh andLevoy [15], to allow multiple rendering threads to coop-eratively generate partial images that are later combinedvia compositing into a final image [6], [7], [14].

This design approach, which uses a blend of object-and pixel-level parallelism, has proven successful inachieving scalability and tackling large data sizes. TheTREX system [3] is a parallel volume rendering algo-rithm on a shared-memory platform that uses object-parallel data domain decomposition and texture-based,hardware-accelerated rendering followed by a paral-lel, software-based composition phase with image-spacepartitioning. The design choices for which part of theSGI Origin to use for different portions of the algorithmreflect a desire to achieve optimal performance at eachalgorithmic stage and to minimize inter-stage communi-cation costs. Muller et al. [16] implemented a raycasting

volume renderer with object-order partitioning on asmall 8-node GPU cluster using programmable shaders.This system sustained frame rates in the single digitsfor datasets as large as 12603 with an image size of1024×768. More recently, Moloney et al. [17] implementa GLSL texture-based volume renderer that runs on a32-node GPU system. This work takes advantage of thesort-first architecture to accelerate certain types of ren-dering, like occlusion culling. Childs et al. [18] present aparallel volume rendering scheme for massive datasets(with one hundred million unstructured elements anda 30003 rectilinear data set). Their approach parallelizesover both input data elements and output pixels, and isdemonstrated to scale well on up to 400 processors.

Peterka et al. [4] run a parallel volume renderingalgorithm at massive concurrency, rendering 44803 datasizes with 32,000 cores on an IBM BG/P system. Theydemonstrated generally good scalability and found thatthe compositing phase slowed down when more than tenthousand cores were involved, likely due to hardware orMPI limitations. To address this problem, they reducedthe number of processors involved in the compositingphase. Later, Peterka et al. introduce the radix-k paral-lel compositing algorithm [19] to address compositingperformance at large concurrency. Kendall et al. [20]build upon this work by demonstrating an approachfor tuning radix-k’s algorithmic parameters to variousarchitectures, and include new compression and loadbalancing optimizations for better scalability and perfor-mance on diverse architectures.

The most substantial difference between our work andprevious work in parallel volume rendering is that weare exploiting hybrid parallelism at extreme concurrency,and performing in-depth studies to better understandscalability characteristics as well as potential perfor-mance gains of the hybrid-parallel approach.

2.2 Hybrid Parallelism

Hybrid parallelism has evolved in response tothe widespread deployment of multi-core chips indistributed-memory systems. The hybrid model allowsdata movement among nodes using traditional MPImotifs like scatter and gather, but within nodesusing shared-memory parallelism via threadedframeworks like POSIX threads or OpenMP. Previouswork comparing distributed-memory versus hybrid-memory implementations (e.g., [21], [22]) has focusedon benchmarking well-known computational kernels.In contrast, our study examines this space from theperspective of visualization algorithms.

The previous studies point to several areas wherehybrid memory may outperform distributed memory.First, hybrid memory tends to require a smaller datafootprint for applications with domain decomposition(e.g. parallel volume rendering), since fewer domainsmeans less “surface area” between domains and henceless exchange of “ghost” data. Second, the MPI runtime

3

allocates various tables, buffers, and constants on a per-task basis. Today, the gain from using fewer tasks toreduce this memory overhead may seem small withonly four or six cores per chip, but the trend towardshundreds of cores per chip with less memory per corewill amplify these gains. Third, hybrid-memory imple-mentations can use a single MPI task per node for collec-tive operations like scatter-gather and all-to-all, therebyreducing the absolute number of messages traversingthe inter-connect. While the size of the messages in thisscenario may be larger or smaller depending upon thespecific problem, a significant factor influencing overallcommunication performance is latency, the cost of whichis reduced by using fewer messages.

Peterka et al. [23] investigated a hybrid-parallel im-plementation of a volume renderer that uses four POSIXthreads per MPI task on the IBM BlueGene/P archi-tecture. They identify similar improvements during thecompositing phase as we report in our study. Theirstrong scaling study extends to 4096-way concurrencywith a 11203 dataset and an 12042 image, whereas weconduct both strong and weak scaling studies withup to a 230403 dataset and 216,000-way concurrency,and compare hybrid-memory implementations that usePOSIX threads, OpenMP, and CUDA.

Fogal et al. [24] have recently studied the performanceof volume rendering on a larger multi-GPU cluster.While their use of an MPI-based compositing phase issimilar to our implementation, they use a different slice-based volume rendering technique implemented withGLSL shaders in OpenGL. Their implementation is anadd-on to VisIt, a production parallel visualization tool,whereas our implementation is a stand-alone researchprototype. Still, their results show similar advantagesfor increasing the density of cores per node, as madepossible by a GPU cluster. Because they use out-of-core access to CPU memory, they are able to processdatasets as large as 81923 on 256 GPUs. In contrast, weoperate within the available device memory of 448 GPUsto process a 46083 dataset. However, we explore imagesizes as large as 46082, while Fogal et al. use a smaller1024× 768 image.

3 IMPLEMENTATION

From a high level view, our parallel volume renderingimplementation using a design pattern that forms thebasis of previous work (e.g., [9]–[12]). Given a sourcedata volume S and n parallel tasks, each task reads in1/n of S, performs raycasting volume rendering on thisdata subdomain to produce a set of image fragments,then participates in a compositing stage in which frag-ments are exchanged and combined into a final image.The completed image is gathered to the root task fordisplay or I/O to storage. Figure 2 provides a block-levelview of this organization.

Our distributed-memory implementation is writtenin C/C++ using the MPI [25] library. The portions

of the implementation that are shared-memory parallelare written using a combination of C/C++ and andeither POSIX threads [26], OpenMP [27], or CUDA (ver-sion 3.0) [28] so that we are actually comparing threehybrid-memory implementations that we refer to ashybrid/pthreads, hybrid/OpenMP, and hybrid/CUDA.

The distributed-memory and hybrid-memory imple-mentations differ in several key respects. First, theraycasting volume rendering algorithm runs in serialon the distributed-memory tasks, but is multi-threadedon the hybrid-memory tasks. We discuss this issue inmore detail in Section 3.1. Second, the communicationtopology in the compositing stage differs slightly, andwe discuss this issue in more detail in Section 3.2. Athird difference is in how data is partitioned across thetasks. In the distributed-memory implementation, eachtask loads and operates on a disjoint block of data. Inthe hybrid-memory implementation, each task loads adisjoint block of data and each of its worker threadsoperate in parallel on that data using an image-paralleldecomposition [15].

3.1 Parallel RaycastingOur raycasting volume rendering code implementsLevoy’s method [6]: we compute the intersection of aray with a data block, and then compute color at afixed step size along the ray through the volume. Allcolors along the ray are composited front-to-back usingthe “over” operator. Output consists of a set of imagefragments that contain an x, y pixel location, R,G,B, αcolor, and a z-coordinate. The z-coordinate is the locationin eye coordinates where the ray penetrates the block ofdata. Later, these fragments are composited in the correctorder to produce a final image (see Section 3.2).

Each distributed-memory task invokes a serial ray-caster that operates on its own disjoint block of data.Since we are processing structured rectilinear grids, alldata subdomains are spatially disjoint, so we can safelyuse the ray’s entry point into the data block as the z-coordinate for sorting during the subsequent composi-tion step.

In contrast, the hybrid-memory tasks invoke a ray-caster with shared-memory parallelism that consists of Tthreads executing concurrently to perform the raycastingon a shared block of data. As in [15], we use an image-space partitioning: each thread is responsible for raycast-ing a portion of the image. In the pthreads and OpenMPraycasting implementation, our image-space partitioningis interleaved, with the image divided into many tilesthat are distributed amongst the threads. The CUDAraycasting implementation is slightly different becauseof the data-parallel nature of the language: the imageis treated as a 2D CUDA grid, which is divided intoCUDA thread blocks. Each thread block corresponds toan image tile, and the individual CUDA threads withineach block are mapped to individual pixels in the image.

We directly ported our pthreads raycasting implemen-tation to CUDA and made no additional optimizations to

4

Fig. 2. Diagram of our system architecture.

improve performance. Even though CUDA thread blocksrequire a minimum of 32 threads to saturate computa-tional throughput, our optimal configuration uses only 8threads, which points to the branching nature of our al-gorithm that causes “divergence” among CUDA threads.A thread block is executed in a single-instruction-multiple-thread (SIMT) fashion in which “warps” of32 threads are executed across 4 clock cycles in setsof 8 threads that share a common instruction. If those8 threads do not share a common instruction, suchas when conditionals cause branching code paths, thethreads “diverge” and must be executed individually.This situation is prevalent in our algorithm. For example,imagine a thread block owning a region of the imagethat only partially covers the data volume. Some of thethreads immediately exit because of the “empty-space-skipping” [16] optimization in our algorithm, while theother threads proceed to cast rays through the volume.Even the threads that proceed together with raycastinghave rays of different lengths, which can cause diver-gence and load imbalance.

It is possible to improve performance of the CUDAraycasting implementation by dividing it into separatestages for calculating the ray intersections and integrat-ing the rays, but the trade-off is the additional memoryrequired to temporarily buffer the rays between the twostages. Although we tested this optimization and it ledto better performance at small scales, we chose to usethe direct CUDA raycasting port in our study so as tomaintain a algorithmically consistent comparison withthe pthreads and OpenMP implementations.

3.2 Parallel Compositing

Compositing begins by partitioning the pixels of the finalimage across the tasks. Next, an all-to-all communicationstep exchanges each fragment from the task where itwas generated in the raycasting phase to the task thatowns the region of the image in which the fragmentlies. This exchange is done using an MPI_Alltoallvcall. After the exchange, each task then performs the

final compositing for each pixel in its region of theimage using the “over” operator, and the final imageis gathered on the root task.MPI_Alltoallv provides the same functionality as

performing direct sends and receives, but bundles themessages into fewer point-to-point exchanges for greaterefficiency. This call uses a variety of strategies basedon concurrency level and total message size. One ofthe strategies employed is pairwise exchange, so, pro-vided the heuristics of which strategy to employ aregood, the MPI_Alltoallv should always match orexceed that of pairwise exchange. Pairwise exchangeshares certain characteristics with binary swap, namelymessages are collated and exchanged in a way thatminimizes the number of point to point communications.The key difference between binary swap and pairwiseexchange is that binary swap calculates intermediatecompositings as messages are exchanged between thetasks, reducing the message sizes as as the algorithmprogresses. Although our implementation may appearto be related to direct-send at first glance, it in facthas performance characteristics closer to binary swap,with the caveat that the total message size is greater.Of course, an optimized binary swap implementationwould outperform our implementation and the resultsfrom Peterka et al in [4] showed the Radix-K providedfurther improvement, improving on binary swap byas much as 40%. Regardless, we believe the numberof messages and message sizes for MPI_Alltoallv issimilar to those of the more sophisticated algorithmsand therefore findings regarding the benefits of hybridparallelism also provide evidence of potential benefit forthose algorithms.

Peterka et al. [4] reported scaling difficulties for com-positing when using more than 8,000 tasks. They solvedthis problem by reducing the number of tasks receivingfragments to be no more than 2,000. We emulated thisapproach, again limiting the number of tasks receiv-ing fragments, although we experimented with valueshigher than 2,000.

In the hybrid-memory implementations, only one

5

thread per socket participates in the compositing phase.That thread gathers fragments from all other threadsin the same socket, packs them into a single buffer,and transmits them to other compositing tasks. This ap-proach results in fewer messages than if all threads in thehybrid-memory implementation were to send messagesto all other threads. Our aim here is to better under-stand the opportunities for improving performance inthe hybrid-memory implementation. The overall effect ofthis design choice is an improvement in communicationcharacteristics, as indicated in Section 5.4.

4 METHODOLOGY

Our methodology is designed to test the hypothesis thata hybrid-memory implementation exhibits better per-formance and resource utilization than the distributed-memory implementation.

Our multi-core CPU test system, JaguarPF, is a CrayXT5 located at Oak Ridge National Lab and in 2009was ranked number one on the list of Top 500 fastestsupercomputers with a peak theoretical performanceof 2.3 Petaflop [29]. Each of its 18,688 nodes has twosockets, and each socket has a six-core 2.6GHz AMDOpteron processor, for a total of 224,256 compute cores.With 16GB per node (8GB per socket), the system has292TB of aggregate memory and roughly 1.3GB per core.

Our many-core GPU cluster, Longhorn, is located atthe Texas Advanced Computing Center and has 256 hostnodes with dual-socket quad-core Intel Nehalam CPUsand 24GB of memory. They share 128 NVIDIA OptiPlex2200 external quad-GPU enclosures for a total of 512FX5800 GPUs. Each GPU has a clock speed of 1.3Ghz,4GB of device memory, and can execute 30 CUDA threadblocks concurrently. We choose to treat the FX5800 asa generic “many-core” processor with a data-parallelprogramming model (CUDA) that serves as a surrogatefor anticipating what future many-core clusters may looklike. In terms of performance, we position the FX5800relative to the Opteron in terms of its actual observedruntime for our particular application rather than relyingon an a priori architectural comparison.

We conducted the following three studies:• strong scaling, in which we fixed the image size at

46082 and the dataset size at 46083 (97.8 billion cells)for all concurrency levels;

• weak–dataset scaling, with the same fixed 46082

image, but a dataset size increasing with concur-rency up to 230403 (12.2 trillion cells) at 216,000-wayparallel; and

• weak scaling, in which we increased both the imageand dataset up to 230402 and 230403, respectfully, at216,000-way parallel.

Note that at the lowest concurrency level, all three casescoincide with a 46082 image and 46083 dataset size.

On JaguarPF, we share a data block among six threadsand use one sixth as many tasks. Although we couldhave shared a data block among as many as twelve

threads on each dual-socket six-core node, sharing dataacross sockets results in non-uniform memory access.Based on preliminary tests, we estimated this penaltyto be around 5 or 10% of the raycasting time. Therefore,we used six threads running on the cores of a single six-core processor. On Longhorn, we use one task per GPUdevice, and load the data block into GPU device memorywhere it is shared among all CUDA threads.

Because the time to render is view-dependent, weexecuted each raycasting phase ten times over a selectionof ten different camera locations. The raycasting timeswe report are an average over all locations.

In the compositing phase, we tested either four orfive (depending on memory constraints) ratios of totaltasks to compositing tasks. We restricted the compositingexperiment to only two views (the first and last) becauseit was too costly to run a complete battery of all view andratio permutations. Since the runtime of each trial canvary due to contention for JaguarPF’s interconnectionfabric with other users, we ran the compositing phaseten times for both views. We report mean and minimumtimes over this set of twenty trials. Minimum times mostaccurately represent what the system is capable of underoptimal, contention-free conditions, while mean timeshelp characterize the variability of the trials.

4.1 Determining Optimal Algorithmic ParametersThe size and shape of the image tiles used in an image-space partitioning for shared-memory raycasting canbe treated as a tunable parameter. To determine whichconfigurations performed best on the AMD OpteronCPU, we ran a parameter sweep over 100 differentsizes and shapes using our shared-memory raycaster ona 3843 dataset on a single CPU with six threads. Wevaried the image tile width and height over the range{1, 2, 4, . . . , 256, 512} and measured frame rendering time(FRT) over thirty different views using dynamic workassignments. For this problem, there are three “invalid”block size configurations, 256 × 512, 512 × 512, and512×256, that produce decompositions resulting in onlyone or two total work blocks.

From the test results, shown in Figure 3, we see clear“sweet” and “sour” spots in performance. To producethat figure, we “normalized” FRT by the minimum sothat values range from [1.0 . . .∞]. In this particular setof results, the maximum data value is about 3.0, whichmeans the FRT of the worst-performing tile size wasabout three times as slow as the best-performing con-figuration. We see that very small and very large imagetiles produce poorer performance: at small tile sizes, thethreads’ access to the work list is serialized through amutex and performance is adversely affected. At largetile sizes, there are not enough tiles to result in goodload balance. Therefore, the sweet spot tends to fall inregions of medium-sized tiles.

An interesting feature in Figure 3 is the relatively poorperformance for block widths of {1, 2, 4}. At these config-urations, we observe in hardware performance counter

6

Fig. 3. Normalized frame rates resulting from usingdifferent image tile sizes in the pthreads implementation.Reds correspond to higher and blues to lower frame rates.

CUDA Block Size

1 2 4 8

X Dimension

1

2

4

8

Y D

imen

sion

1.2

1.4

1.6

1.8

2

2.2

2.4

2.6

2.8

3

3.2

Avg. Raycasting Tim

e (s)

Fig. 4. A sweep over CUDA thread block sizes found that4× 2 was the optimal configuration.

data a relatively high level of L1, L2, and TLB cachemisses (not shown due to space limitations); there seemsto be a direct correlation between higher cache miss ratesand higher FRT. The reason for higher cache miss ratesin general is due to degraded spatial and/or tempo-ral locality of memory accesses. We are using a row-major ordering of volume data in memory for each datablock, which could be a contributing factor to relativelyhigher cache miss rates in certain configurations. Morestudy, including comparison with more cache-friendlylayout schemes like Z-ordering [30], would help revealmore insights into the cache utilization characteristicsof different parameter choices. Nonetheless, the mainpoint here is that our approach, which is consistentwith autotuning in general, is to empirically measureperformance across a range of algorithmic parametersin order to determine the configurations that perform

better and worse for a given problem configuration on agiven platform. The alternative is to attempt to derive aquantitative performance model of a complex system inorder to predict which parameter settings will result inthe best performance given a specific algorithm, problemconfiguration, and machine architecture.

Based upon the results of this study, we decided toperform our scalability studies using an image tile sizeof 32×32 pixels. That tile size lies within the sweet spotthat is visible in Figure 3. Other tile size choices couldbe equally valid: we see that tile sizes having relativelylarge width and relatively small height also lie withinthe sweet spot of FRT.

We also performed a sweep over CUDA thread blocksizes on the FX5800 using the MPI+CUDA implemen-tation on a subset of 48 GPUs (see Figure 4). For theFX5800, the maximum number of CUDA threads perthread block is 512, and all registers for all threads inthe block must fit in only 16KB of memory. With theseconstraints, we swept over thread block sizes up to 8×8.The experiment found that 4× 2 was the optimal size.

On the GPU, image tiles (CUDA thread blocks) areautomatically load balanced by the CUDA runtime’sscheduler. On the CPU, however, we had the choice ofimplementing either a dynamic or static work assign-ment of image tiles to threads. We ran a similar studywhere we varied tile size across both algorithms. Theresults of that study, not shown here due to space limi-tations, indicate the static assignment produces slightlybetter frame rates at the smallest block sizes, but that thedynamic assignment produces better results at mediumand larger block sizes and has better load balance char-acteristics across a larger range of block sizes. Therefore,we chose to use the dynamic work assignment since itresults in better performance than static assignments for32× 32 tiles.

4.2 Source Data and Decomposition

Starting with a 5123 dataset of combustion simulationresults 1, we used trilinear interpolation to upscale itto arbitrary sizes in memory. We scaled equally in allthree dimensions to maintain a cubic volume. Our goalwas to choose a problem size that came close to fillingall available memory (see Table 1). Although upscalingmay distort the results for a data-dependent algorithm,the only data dependency during raycasting is early raytermination. However, we found that for our particulardataset and transfer function, there was always at leastone data block for which no early terminations occurred.Moreover, the cost of the extra conditional statement in-side the ray integration loop to test for early terminationadded a 5% overhead. Therefore, we ran our study withearly ray termination turned off, and we believe thatupscaling the dataset does not effect our results.

1. Sample data courtesy J. Bell and M. Day, Center for ComputationalSciences and Engineering, Lawrence Berkeley National Laboratory.

7

TABLE 1Problem Configurations

Strong Scaling Weak Scaling

Distributed Tasks Hybrid Tasks Distributed Block Hybrid Block Memory Per GPU/Node (MB) Data Size

GPU

- 56 - 1152× 576× 329 3331 2304× 2304× 2303

- 112 - 576× 576× 329 1666 2900× 2900× 2898

- 224 - 576× 288× 329 833 3656× 3656× 3654

- 448 - 288× 288× 329 416 4608× 4608× 4606

CPU

1728 (123) 288 384× 384× 384 384× 768× 1152 10368 46083

13824 (243) 2304 192× 192× 192 192× 384× 576 1296 92163

46656 (363) 7776 128× 128× 128 128× 256× 384 384 138243

110592 (483) 18432 96× 96× 96 96× 192× 288 162 184323

216000 (603) 36000 76× 76× 76 76× 153× 230 80.4/81.6 230403

Because the compute nodes on an XT5 system haveno physical disk for swap space, allocating beyond theamount of physical memory causes program termina-tion. Our total memory footprint was four times thenumber of bytes in the entire dataset: one for the datasetitself, and the other three for the gradient data, whichwe computed by central difference and used in shadingcalculations. Although each node has 16GB of memory,we could reliably allocate only 10.4GB for the data blockand gradient field at 1,728-way concurrency because ofoverhead from the operating system and MPI runtimelibrary.

On JaguarPF, we chose concurrencies that are cubicnumbers to allow for a clean decomposition of the entirevolume into cubic blocks per distributed-memory task.In hybrid memory, these blocks are rectangular becausewe aggregated six blocks (1×2×3) into one shared block.

4.2.1 Strong ScalingOn JaguarPF, we used a fixed 46083 dataset at allconcurrencies, except for 216,000-way where the 46083

dataset could not be evenly divided. With increasingconcurrency, each task was assigned a correspondinglysmaller data block, as seen in Table 1.

At 216,000-way concurrency we rounded down to a45603 distributed-memory dataset and a 4560 × 4590 ×4600 hybrid-memory dataset. As a result, the distributed-memory dataset is approximately 1.4% smaller. Whilethis difference might seem to give an advantage tothe distributed-memory implementation, results in latersections show that hybrid-memory performance andresource utilization are uniformly better.

On Longhorn, we were unable to use all 512 GPUs,so we instead configured our largest test as 8 × 8 × 7blocks over 448 GPUs. Starting at 56-way concurrency,the largest dataset we could reliably fit in GPU devicememory was 23043. We processed this data size witha matching image size of 23042 at concurrencies in therange {56, 112, 224, 448}.

4.2.2 Weak–dataset and Weak ScalingOn JaguarPF, we assigned a 3843 block to eachdistributed-memory task t all concurrency levels, and

aggregated these blocks into a 384 × 786 × 1152 blockfor the hybrid-memory tasks. As a result, we processeddatasets with sizes ranging from 46083 to 230403 (seeTable 1). For the weak–dataset scaling, we maintainedthe same 46082 image size at all concurrency levels,whereas for weak scaling we scaled the image size upto 230402.

On Longhorn, we conducted a weak scaling studyfrom 56- to 448-way concurrency with data sizes rangingfrom 23043 to 46083 and image sizes from 23042 to46082. We had to round down the z-dimensions of thedataset slightly to maintain integer multiples in thedecomposition (see ‘Data Size’ in Table 1).

5 RESULTS

We compare the cost of MPI runtime overhead and cor-responding memory footprint in Section 5.1; the absoluteamount of memory required for data blocks and ghost(halo) exchange in Section 5.2; the scalability of the ray-casting and compositing algorithms in Sections 5.3 and5.4; and the communication resources required duringthe compositing phase in Section 5.4. We conclude inSection 5.5 with a comparison of results from the six-core CPU system and the many-core GPU system tounderstand how the balance of shared-memory versusdistributed-memory parallelism affects overall perfor-mance.

5.1 InitializationBecause there are fewer hybrid-memory tasks, they in-cur a smaller aggregate memory footprint for the MPIruntime environment and program-specific data struc-tures that are allocated per task. Table 2 shows thememory footprint of the program as measured directlyafter calling MPI_Init and reading in command-lineparameters 2. Memory usage was sampled only fromtasks 0 through 6, but those values agreed within 2% ofeach other. Therefore, the per-task values we report inTable 2 are from task 0 and the per-node and aggregatevalues are calculated from the per-task value.

2. We collected the VmRSS, or “resident set size,” value from the/proc/self/status interface.

8

TABLE 2Memory Usage at MPI Initialization

CPUCores

MemoryType

Tasks Per Proc.(MB)

Per Node(MB)

Agg.(GB)

1728 Hybrid 288 67 133 191728 Distrib. 1728 67 807 113

13824 Hybrid 2304 67 134 15113824 Distrib. 13824 71 857 96546656 Hybrid 7776 68 136 51846656 Distrib. 46656 88 1055 4007

110592 Hybrid 18432 73 146 1318110592 Distrib. 110592 121 1453 13078216000 Hybrid 36000 82 165 2892216000 Distrib. 216000 176 2106 37023

TABLE 3CUDA Initialization Time (s)

ConcurrencyScaling 56 112 224 448Strong 2.10 2.06 1.83 1.73Weak 2.10 2.15 2.53 2.47

The distributed-memory implementation uses twelvetasks per node while the hybrid-memory one uses onlytwo. At 216,000-way concurrency, the runtime over-head per distributed-memory task is more than 2× theoverhead per hybrid-memory task. The per-node andaggregate memory usage is another factor of six largerfor the distributed-memory implementation because ituses 6× as many tasks. Thus, the distributed-memoryimplementation uses nearly 12× as much memory per-node and in-aggregate than the hybrid-memory imple-mentation for initializing the MPI runtime at 216,000-way concurrency.

A major disadvantage of using GPU co-acceleratorsis the cost of initializing the GPU device in CUDA andof transferring data between host and device memories.For our hybrid/CUDA implementation, it took up to2.53s to initialize the GPU on each node and transfer thedata block from CPU memory to GPU memory acrossthe PCI-Express bus (transferring the array of imagefragments back to CPU memory had negligible cost).Although this upfront cost can be amortized in the usecase where several volume renderings are computedfor the same dataset, if only one volume rendering iscomputed, then this cost makes the hybrid/CUDA im-plementation less competitive than the hybrid/pthreadsor hybrid/OpenMP ones. However, the initializationcost is a limitation of the hybrid/CUDA implementationin particular, and not the hybrid-parallelism approach ingeneral.

5.2 Ghost DataTwo layers of ghost data are required in our raycast-ing phase: the first layer for trilinear interpolation ofsampled values, and the second layer for computing thegradient field using central differences (gradients are not

0

20

40

60

1728 13824

46656

110592

216000

Gho

st D

ata

(GB)

Strong Scaling

DistributedHybrid

0

400

800

1200

1600

1728 13824

46656

110592

216000

Gho

st D

ata

(GB)

CPU Cores

Weak Scaling

Fig. 5. Ghost data requirements.

pre-computed for our dataset). Because hybrid memoryis decomposed into fewer, larger blocks, it requires lessexchange and storage of ghost data by roughly 40%across all concurrency levels and for both strong andweak scaling (see Figure 5).

5.3 RaycastingAll of the scaling studies demonstrated good scaling forthe raycasting phase, as no message passing is involved(see Figure 6). For these runs and timings, we usedtrilinear interpolation for data sampling along the rayas well as a Phong-style shader. The final raycastingtime is essentially the runtime of the thread that takesthe most integration steps. This behavior is entirelydependent on the view. Our approach, which is aimedat understanding “average” behavior, uses ten differentviews and reports an average runtime.

In the strong scaling study, we achieved linear scalingup to 216,000-way concurrency for the raycasting phasewith distributed memory (see Figure 6). The hybrid-memory implementation exhibited different scaling be-havior because of its different decomposition geometry:the distributed blocks had a perfectly cubic decomposi-tion, while in hybrid memory we aggregated 1 × 2 × 3cubic blocks into a larger rectangular block (see Table1). The smaller size of the GPU cluster limited theconcurrencies we could choose for the hybrid/CUDAimplementation, leading to similarly irregular blocks inthat case.

The interaction of the decomposition geometry andthe camera direction determines the maximum numberof ray integration steps, which is the limiting factor forthe raycasting time. At lower concurrencies, this inter-action benefited the hybrid-memory implementation by

9

TABLE 4Raycasting Time (s)

Strong Scaling Weak–dataset Scaling Weak Scaling Hybrid/CUDA

Cores Distributed Hybrid/ pthreads Hybrid/ OpenMP ” ” ” ” ” ” GPUs Strong Weak1728 24.88 22.31 22.23 24.88 22.31 22.23 24.88 22.31 22.23 56 1.04 1.04

13824 3.10 2.84 2.83 7.33 7.06 7.06 26.65 24.92 24.93 112 0.54 1.0946656 0.92 0.85 0.85 3.56 3.51 3.51 27.18 25.88 25.88 224 0.29 1.16

110592 0.38 0.37 0.37 2.13 2.15 2.15 27.45 26.59 26.50 448 0.15 1.18216000 0.19 0.21 0.20 1.46 1.48 1.58 *22.17 26.79 26.88

* Data size is smaller due to memory constraints.

1

2

4

8

27

64

125

1728 13824

46656

110592

216000

56 112 224 448

Ray

cast

ing

Spee

dup

CPU Cores

Strong ScalingGPUs

Linear ScalingDistributed

Hybrid/pthreadsHybrid/OpenMP

Hybrid/CUDA

1

2

4

8

16

32

1728 13824

46656

110592

216000

Ray

cast

ing

Spee

dup

CPU Cores

Weak-dataset Scaling

Linear ScalingConstant Scaling

DistributedHybrid/pthreadsHybrid/OpenMP

0

0.2

0.4

0.6

0.8

1

1728 13824

46656

110592

216000

56 112 224 448

Ray

cast

ing

Spee

dup

CPU Cores

Weak ScalingGPUs

*

Constant ScalingDistributed (measured)Distributed (estimated)

Hybrid/pthreadsHybrid/OpenMP

Hybrid/CUDA

Fig. 6. The speedups (referenced to 1,728 cores) for both the raycasting phase and the total render time (raycastingand compositing). The raycasting speedup is linear with distributed memory, but sublinear with hybrid memory: thiseffect is caused by the difference in decomposition geometries (cubic versus rectangular).

as much as 11% (see Table 4). At higher concurrenciesthe trend flips and the distributed-memory implemen-tation outperforms the hybrid-memory one by 10%. Weexpect that if we were able to run the hybrid-memoryimplementation with regular blocks (such as 2×2×2 onan eight-core system), both implementations would scaleidentically. We also note that at 216,000 cores, raycastingis less than 20% of the total runtime (see Figure 9),and the hybrid-memory implementation is over 50%faster because of gains in the compositing phase thatwe describe in the next subsection.

For weak scaling, the hybrid-memory implementationmaintained 80% scalability out to 216,000 cores. Overallraycasting performance is only as fast as the slowestthread and because of perspective projection, the numberof samples each thread must calculate varies. This vari-ation becomes larger at scale as each core is operatingon a smaller portion of the overall view frustum, whichaccounts for the 20% degradation.

The distributed-memory result at 216,000-way con-currency appears (misleadingly) to be superlinear, butthat is because we could not maintain the data sizeper core. Although we could maintain it for the weak–dataset scaling, increasing the image size to 230403 inthe weak scaling study caused the temporary buffers forthe image fragments (the output of the raycasting phase)to overflow. To accommodate the fragment buffer, we

had to scale down to a data size of 192003 (7.1 trillioncells), instead of the 230403 data size (12.2 trillion cells)used in hybrid memory. We report the measured valuesfor this smaller data size, and also estimated values forthe full data size assuming linear scaling by the factor230403/192003.

For the weak–dataset scaling study, the expected scal-ing behavior is neither linear nor constant, since theamount of work for the raycasting phase is dependent onboth data size and image size. With a varying data sizebut fixed image size, the scaling curve for weak–datasetscaling should lie between those of the pure weak andpure strong scaling, which is what we observe. Overall,216,000-way concurrency was 10× faster than 1,728-wayconcurrency.

5.4 CompositingAbove 1,728-way concurrency, we observed that com-positing times are systematically better for the hybrid-memory implementation (see Figure 7). The com-positing phase has two communication costs: theMPI_Alltoallv call that exchanges fragments fromthe task where they originated during raycasting to thecompositing task that owns their region of image space;and the MPI_Reduce call that reduces the final imagecomponents to the root task for assembly and outputto a file or display (see Figure 9 for a breakdown of

10

0

1

2

3

4

10 100 1000 10000 100000

Tim

e in

sec

onds

Strong Scaling

0

1

2

3

4

10 100 1000 10000 100000

Tim

e in

sec

onds

0

1

2

3

4

10 100 1000 10000 100000

Tim

e in

sec

onds

0

1

2

3

4

10 100 1000 10000 100000

Tim

e in

sec

onds

0

1

2

3

4

10 100 1000 10000 100000

Tim

e in

sec

onds

Log10(Compositors)

0

1

2

3

4

10 100 1000 10000 100000

Weak-dataset Scaling

0

1

2

3

4

10 100 1000 10000 100000

0

1

2

3

4

10 100 1000 10000 100000

0

1

2

3

4

10 100 1000 10000 100000

0

1

2

3

4

10 100 1000 10000 100000Log10(Compositors)

0

1

2

3

4

10 100 1000 10000 100000

1,728Weak Scaling

DistributedHybrid/OpenMPHybrid/pthreads

0

1

2

3

4

5

6

7

8

10 100 1000 10000 100000

13,824

0 1 2 3 4 5 6 7 8 9

10 11 12

10 100 1000 10000 100000

46,656

0 1 2 3 4 5 6 7 8 9

10 11 12

10 100 1000 10000 100000

110,592

0 1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16

10 100 1000 10000 100000

216,00

Log10(Compositors)

Fig. 7. Compositing times for different ratios of compositing tasks to total tasks. Solid lines show minimum times takenover ten trials each for two different views; dashed lines show the corresponding mean times.

11

0.0e+00

2.0e+05

4.0e+05

6.0e+05

8.0e+05

1.0e+06

1.2e+06

1728 13824

46656

110592

216000

Mes

sage

sStrong Scaling

0.0e+00

2.0e+05

4.0e+05

6.0e+05

8.0e+05

1.0e+06

1.2e+06

1.4e+06

1.6e+06

1728 13824

46656

110592

216000

Weak-dataset Scaling

0.0e+00

5.0e+05

1.0e+06

1.5e+06

2.0e+06

2.5e+06

3.0e+06

3.5e+06

4.0e+06

1728 13824

46656

110592

216000

Weak ScalingDistributed

Hybrid/OpenMPHybrid/pthreads

0

5

10

15

20

25

1728 13824

46656

110592

216000

Dat

a Se

nt (G

B)

CPU Cores

0

5

10

15

20

25

1728 13824

46656

110592

216000

CPU Cores

0

100

200

300

400

500

600

700

1728 13824

46656

110592

216000

CPU Cores

Fig. 8. The number of messages and total data sent during the fragment exchange in the compositing phase.

these costs). During the fragment exchange, the hybrid-memory implementation can aggregate the fragmentsin the memory shared by six threads, and thereforeit uses on average about 6× fewer messages than thedistributed-memory implementation (see Figure 8). Inaddition, the hybrid-memory implementation exchangesless fragment data because its larger data blocks allowfor more compositing to take place during ray integra-tion. Similarly, one sixth as many hybrid-memory tasksparticipate in the image reduction, which leads to betterperformance.

Our compositing phase used a subset of tasks for com-positing and we explored different ratios of compositorsto renderers. For the strong and weak–dataset scalingstudies, we observe an inflection point starting at 13,824-way concurrency for hybrid memory and 46,656-wayfor distributed memory, where the optimal compositingconfiguration is to use a subset in the range of 2,000 to8,000 tasks. We believe this inflection point exists due tothe characteristics of the underlying interconnect fabric.Because the hybrid-memory implementation generatesfewer and smaller messages, the critical point occurs at ahigher level of concurrency than for distributed memory.Interestingly, in the weak scaling study, the inflectionpoint does not arise until higher concurrency, and usinga larger subset of compositing tasks in the range of 2,000to 30,000 is optimal. Moreover, it was not possible to usesubsets as small as in the strong scaling study becauseof the larger image size with weak scaling; setting thesubset too small led to overallocation from temporarydata structures and MPI buffers as the additional datawas concentrated in fewer compositors.

Peterka et al. [4] first observed that the reduced num-ber of compositors at higher concurrencies increasedoverall performance. In their strong scaling study, theyconcluded that 1,000 to 2,000 compositors were optimal

for up to 32,768 total tasks. We note, however, that thereare many differences between our study and theirs inthe levels of concurrency, architectures, operating sys-tems, communication networks, and MPI libraries, eachpotentially introducing variation in the ideal number ofcompositors.

5.5 Overall PerformanceIn the strong scaling study at 216,000-way concurrency,the best compositing time with hybrid memory (0.35s,4500 compositors) was 3× faster than with distributed(1.06s, 6750 compositors). Furthermore, at this scalecompositing time dominated rendering time, which wasroughly 0.2s for both implementations. Thus, the totalrender time was 2.2× faster with hybrid memory (0.56sversus 1.25s). Overall, the strong scaling study showsthat the advantage of hybrid memory over distributedbecomes greater as the number of cores increases (seeFigure 9).

For weak–dataset and weak scaling, the hybrid-memory implementation still shows gains over thedistributed-memory one, but they are less pronouncedbecause the raycasting phase dominates. Although the216,000-way breakdown for weak scaling looks like itfavors distributed memory, this is actually an artifact ofthe reduced data size (192003) we were forced to usein the distributed-memory implementation to avoid out-of-memory errors. Comparing an estimated value for a230403 distributed-memory data size suggests that thehybrid-memory implementation would be slightly faster.

At 448-way concurrency with the 46083 dataset, thehybrid/CUDA implementation averaged 1.18s for ray-casting and exhibited a minimum compositing time of0.15s (median of 0.19s). The raycasting performance po-sitions this run close to the 46,656-way run on JaguarPF(see Figure 10). However, the compositing time for the

12

0

20

40

60

80

100

Strong Scaling 1,728 13,824 46,656 110,592 216,000

25.322.9 22.8

3.6

3.2 3.2

1.5

1.2 1.2

1.2

0.7 0.7

1.2

0.6 0.6

0

20

40

60

80

100

Perc

ent o

f "D

istri

bute

d" T

ime W

eak-data Scaling

25.322.9 22.8

7.97.4 7.4

4.13.8 3.8

2.9

2.5 2.5

2.6

1.9 1.9

0

20

40

60

80

100

Distributed

Hybrid/OpenMP

Hybrid/pthreads

Distributed

Hybrid/OpenMP

Hybrid/pthreads

Distributed

Hybrid/OpenMP

Hybrid/pthreads

Distributed

Hybrid/OpenMP

Hybrid/pthreads

Distributed

Hybrid/OpenMP

Hybrid/pthreads

W

eak Scaling

*

* Estimated

25.322.9 22.8

28.126.1 26.1

29.828.1 28.0

32.029.9 29.9

34.0*32.0 31.8

Raycasting Compositing - Alltoallv Compositing - Reduce Compositing - Other

Fig. 9. Total render time split into raycasting and compositing components and normalized at each concurrency levelas a percentage of the distributed-memory time. “Compositing – Other” includes the time to coordinate the destinationsfor the fragments and to perform the over operator.

hybrid/CUDA run on Longhorn (0.15s) is half thatof hybrid-memory runs at 46,656-way concurrency onJaguarPF (0.31 to 0.33s).

This result from Longhorn points more generally tothe increasing potential of hybrid parallelism for raycast-ing volume rendering at higher core-to-node ratios. Theshared-memory parallelism used by the dynamically-scheduled, image-based decomposition in the raycastingphase scales well to high concurrencies per node (thou-sands of threads in the case of CUDA). In turn, decreas-ing the absolute number of nodes improves the commu-nication performance during the compositing phase.

We anticipate that future multi-core CPU systems withmore shared-memory parallelism will find a similar“sweet spot” in hybrid-parallel volume rendering per-formance as we observed on Longhorn. For instance, weexpect that our hybrid/pthreads and hybrid/OpenMPimplementations will benefit from additional shared-memory parallelism on the recently procured Cray XT5

system Hopper at NERSC, which will have dual-socket12-core Opteron CPUs for 24 cores per node.

6 CONCLUSION AND FUTURE WORKThe multi-core era offers new opportunities and chal-lenges for parallel applications. Our study shows thatfor raycasting volume rendering, hybrid parallelism en-ables performance gains and uses less resources than adistributed-memory implementation on large supercom-puters comprised of both multi-core CPUs and many-core GPUs. The advantages are reduced memory foot-print, reduced MPI overhead, and reduced communica-tion traffic. These advantages are likely to become morepronounced in the future as the number of cores per CPUincreases while per-core memory size and bandwidthdecrease.

Our study used MPI_Alltoallv to perform com-positing. While we are encouraged by the results show-ing favorable hybrid-parallel performance in terms of

13

0

1

2

3

4

5

Distributed

Hybrid/OpenMP

Hybrid/pthreads

Hybrid/CUDA

Distributed

Hybrid/OpenMP

Hybrid/pthreads

Tota

l Ren

der T

ime

(s)

JaguarPF 13,824

Longhorn 448JaguarPF 46,656

3.63.2 3.2

1.331.5

1.2 1.2

RaycastingComposite - AlltoallvComposite - Reduce

Composite - Other

Fig. 10. Total render time for the 46083 dataset, with thehybrid/CUDA implementation on Longhorn bracketed bythe strong scaling results from JaguarPF.

reduced number of messages, reduced message size,and faster overall runtime, it would interesting to seeimprovement in the context of compositing algorithmslike binary swap and Radix-K. We leave this for futurework.

ACKNOWLEDGMENT

This work was supported by the Director, Office ofScience, Office and Advanced Scientific Computing Re-search, of the U.S. Department of Energy under ContractNo. DE-AC02-05CH11231 through the Scientific Discov-ery through Advanced Computing (SciDAC) program’sVisualization and Analytics Center for Enabling Tech-nologies (VACET).

This research used resources of the National Centerfor Computational Sciences at Oak Ridge National Lab-oratory, which is supported by the Office of Science ofthe U.S. Department of Energy under Contract No. DE-AC05-00OR22725. Additionally, preliminary results forthis work were collected using resources of the NationalEnergy Research Scientific Computing Center (NERSC),which is supported by the Office of Science of the U.S.Department of Energy under Contract No. DE-AC02-05CH11231.

The authors acknowledge the Texas Advanced Com-puting Center (TACC) at The University of Texas atAustin for providing the GPU cluster Longhorn thatcontributed to the research results reported within thispaper.

REFERENCES

[1] K. Asanovic, R. Bodik, B. C. Catanzaro, J. J. Gebis, P. Husbands,K. Keutzer, D. A. Patterson, W. L. Plishker, J. Shalf, S. W.Williams, and K. A. Yelick, “The Landscape of ParallelComputing Research: A View from Berkeley,” EECS Department,University of California, Berkeley, Tech. Rep. UCB/EECS-2006-183, Dec 2006. [Online]. Available: http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-183.html

[2] C. T. Silva, A. E. Kaufman, and C. Pavlakos, “PVR: High-Performance Volume Rendering,” Computing in Science and En-gineering, vol. 3, no. 4, pp. 18–28, 1996.

[3] J. Kniss, P. McCormick, A. McPherson, J. Ahrens, J. Painter,A. Keahey, and C. Hansen, “Interactive Texture-Based VolumeRendering for Large Data Sets,” IEEE Computer Graphics andApplications, July/August 2001.

[4] T. Peterka, H. Yu, R. Ross, K.-L. Ma, and R. Latham, “End-to-endstudy of parallel volume rendering on the ibm blue gene/p,” inProceedings of ICPP’09 Conference, September 2009.

[5] M. Howison, E. W. Bethel, and H. Childs, “MPI-Hybrid Par-allelism for Volume Rendering on Large, Multicore Clusters,”in Proceedings of Eurographics Parallel Visualization and Graphics,Norkopping, Sweden, May 2010.

[6] M. Levoy, “Display of Surfaces from Volume Data,” IEEE Com-puter Graphics and Applications, vol. 8, no. 3, pp. 29–37, May 1988.

[7] R. A. Drebin, L. Carpenter, and P. Hanrahan, “Volume rendering,”SIGGRAPH Computer Graphics, vol. 22, no. 4, pp. 65–74, 1988.

[8] A. Kaufman and K. Mueller, “Overview of Volume Rendering,”in The Visualization Handbook, C. D. Hansen and C. R. Johnson,Eds. Elsevier, 2005, pp. 127–174.

[9] K.-L. Ma, J. S. Painter, C. D. Hansen, and M. F. Krogh, “A DataDistributed, Parallel Algorithm for Ray-Traced Volume Render-ing,” in Proceedings of the 1993 Parallel Rendering Symposium. ACMPress, October 1993, pp. 15–22.

[10] R. Tiwari and T. L. Huntsberger, “A Distributed Memory Al-gorithm for Volume Rendering,” in Scalable High PerformanceComputing Conference, Knoxville, TN, USA, May 1994.

[11] K.-L. Ma, “Parallel volume ray-casting for unstructured-grid dataon distributed-memory architectures,” in PRS ’95: Proceedings ofthe IEEE symposium on Parallel rendering. New York, NY, USA:ACM, 1995, pp. 23–30.

[12] C. Bajaj, I. Ihm, G. Joo, and S. Park, “Parallel ray casting of visiblyhuman on distributed memory architectures,” in VisSym’99 JointEUROGRAPHICS-IEEE TVCG Symposium on Visualization, 1999,pp. 269–276.

[13] P. Sabella, “A Rendering Algorithm for Visualizing 3D ScalarFields,” SIGGRAPH Computer Graphics, vol. 22, no. 4, pp. 51–58,1988.

[14] C. Upson and M. Keeler, “V-buffer: visible volume rendering,”in SIGGRAPH ’88: Proceedings of the 15th annual conference onComputer graphics and interactive techniques. New York, NY, USA:ACM, 1988, pp. 59–64.

[15] J. Nieh and M. Levoy, “Volume Rendering on Scalable Shared-Memory MIMD Architectures,” in Proceedings of the 1992 Workshopon Volume Visualization. ACM SIGGRAPH, October 1992, pp. 17–24.

[16] C. Muller, M. Strengert, and T. Ertl, “Optimized volume raycast-ing for graphics-hardware-based cluster systems,” in Proceedingsof Eurographics Parallel Graphics and Visualization, 2006, pp. 59–66.

[17] B. Moloney, M. Ament, D. Weiskopf, and T. Moller, “Sort FirstParallel Volume Rendering,” IEEE Transactions on Visualization andComputer Graphics, vol. 99, no. PrePrints, 2010.

[18] H. Childs, M. A. Duchaineau, and K.-L. Ma, “A scalable, hybridscheme for volume rendering massive data sets,” in EurographicsSymposium on Parallel Graphics and Visualization, 2006, pp. 153–162.

[19] T. Peterka, D. Goodell, R. Ross, H.-W. Shen, and R. Thakur, “AConfigurable Algorithm for Parallel Image-compositing Applica-tions,” in Supercomputing ’09: Proceedings of the 2009 ACM/IEEEconference on Supercomputing. New York, NY, USA: ACM, 2009,pp. 1–10.

[20] W. Kendall, T. Peterka, J. Huang, H.-W. Shen, and R. Ross,“Accelerating and Benchmarking Radix-k Image Compositing atLarge Scale,” in Proceedings of Eurographics Parallel Visualizationand Graphics, Norkopping, Sweden, May 2010.

[21] G. Hager, G. Jost, and R. Rabenseifner, “Communication Char-acteristics and Hybrid MPI/OpenMP Parallel Programming onClusters of Multi-core SMP Nodes,” in Proceedings of Cray UserGroup Conference, 2009.

[22] D. Mallon, G. Taboada, C. Teijeiro, J. Tourino, B. Fraguela,A. Gomez, R. Doallo, and J. Mourino, “Performance Evalua-tion of MPI, UPC and OpenMP on Multicore Architectures,” in16th European PVM/MPI Users’ Group Meeting, (EuroPVM/MPI’09),September 2009.

[23] T. Peterka, R. Ross, H. Yu, K.-L. Ma, W. Kendall, and J. Huang,“Assessing improvements in the parallel volume rendering

14

pipeline at large scale,” in Proc. SC’08 Ultrascale VisualizationWorkshop, Austin, TX, 2008.

[24] T. Fogal, H. Childs, S. Shankar, J. Kruger, D. Bergeron, andP. Hatcher, “Large data visualization on distributed memorymulti-GPU clusters,” in Proceedings of High Performance Graphics,2010.

[25] M. Snir, S. Otto, S. Huss-Lederman, D. Walker, and J. Dongarra,MPI – The Complete Reference: The MPI Core, 2nd edition. Cam-bridge, MA, USA: MIT Press, 1998.

[26] D. R. Butenhof, Programming with POSIX threads. Boston, MA,USA: Addison-Wesley Longman Publishing Co., Inc., 1997.

[27] R. Chandra, L. Dagum, D. Kohr, D. Maydan, J. McDonald, andR. Menon, Parallel programming in OpenMP. San Francisco, CA,USA: Morgan Kaufmann Publishers Inc., 2001.

[28] NVIDIA Corporation, NVIDIA CUDATM Programming GuideVersion 3.0, 2010, http://developer.nvidia.com/object/cuda 3 0downloads.html.

[29] “The top 500 supercomputers,” 2009, http://www.top500.org.[30] J. K. Lawder and P. J. H. King, “Using Space-filling Curves for

Multi-dimensional Indexing,” in Lecture Notes in Computer Science,2000, pp. 20–35.

PLACEPHOTOHERE

Mark Howison is an application scientist atBrown University’s Center for Computation andVisualization and a computer systems engineerin Lawrence Berkeley National Laboratory’s Vi-sualization Group. His research interests includescientific computing, visualization, graphics, andparallel I/O. Howison has an MS in computer sci-ence from the University of California, Berkeley.Contact him at mhowison@brown.edu.

PLACEPHOTOHERE

E. Wes Bethel is a staff scientist at LawrenceBerkeley National Laboratory, where he con-ducts and leads research, development, and de-ployment activities in high-performance, parallelvisual data exploration algorithms and architec-tures. Bethel has a PhD in computer sciencefrom the University of California, Davis. He’s amember of ACM Siggraph and IEEE. Contacthim at ewbethel@lbl.gov.

PLACEPHOTOHERE

Hank Childs is a computer systems engineerat Lawrence Berkeley National Laboratory and aresearcher at the University of California, Davis.His research interests include parallel visual-ization and production visualization applications.Childs has a PhD in computer science from theUniversity of California at Davis. Contact him athchilds@lbl.gov.