CUDA and GPU Training: Sessions 1 & 2

April 16 & 23, 2012University of Georgia CUDA Teaching Center

UGA CUDA Teaching Center UGA, through the efforts of Professor Thiab Taha,

has been selected by NVIDIA as a 2011-2012 CUDA Teaching Center

Presenters: Jennifer Rouan, Shan-ho Tsai, John Kerry

Visit us at http://cuda.uga.edu

2

Workshop Outline

Introduction to GPUs and CUDA CUDA Programming Concepts Georgia Advanced Computing Resource Center

(GACRC) “My First CUDA Program” – hands-on

programming project

3

A Little Bit of GPU Background

Graphics processing unit (GPU) evolution driven by market demand for high-quality, real-time 3D graphics in computer applications, especially video games

Microsoft’s DirectX 10 API (2006), introduced a geometry shading stage, which demanded an increase in operation rate, particularly floating-point operations

Nvidia’s GeForce 8800 GPU (2006), introduced unified processors, which mapped three separate graphics stages (vertex shading, geometry processing, and pixel processing) to a single array of processors

Scientists recognize the raw performance potential of this hardware and develop General Purpose GPU computing (GPGPU)

4

Logical pipeline:

Physical loop:

Load balancing – make maximum use of the hardware Can dedicate all resources to optimizing one piece of

hardware

Graphics Pipeline

5

pixel processing

vertex shading

geometry processing

Unified Array of Processors

Compute Unified Device Architecture (CUDA)

Nvidia creates CUDA to facilitate the development of parallel programs on GPUs (2007)

The CUDA language is ANSI C extended with very few keywords for labeling data-parallel functions (kernels) and their associated data

Because Nvidia technology benefits from massive economies of scale in the gaming market, CUDA-enabled cards are very inexpensive for the performance they provide

6

Hardware Summary: CPUs and GPUs Central Processing Units (CPUs) are optimized to complete a

large variety of sequential tasks very quickly Graphics Processing Units (GPUs) are optimized to do one

thing: to perform floating point operations on a large amount of data at one time

Compared to CPUs, GPUs dedicate very little chip area to memory in exchange for more computing cores on the chip

7

memory

CPUGPU

Why Program Massively Parallel Processors? Potential to mass-market applications that are

currently considered supercomputing applications (or “superapplications” [Kirk 2010]), such as biology research, image processing, and 3D imaging and visualization

Many of today’s medical imaging applications are still running on microprocessor clusters and special-purpose hardware, and could achieve size and cost improvement on a GPU

Market demand for even better user interfaces and still more realistic gaming is not going to go away

8

Speed Tests on UGA Equipment

Equipment: Barracuda: Nvidia GeForce GTX 480 GPU (480 cores) Z-Cluster: Nvidia Tesla C2075 GPU (448 cores) R-Cluster: Nvidia Tesla S1070 GPU (240 cores) CPU only (serial) : Intel Dual Processor Quad-core Xeon CPU

Testing: Multiply two square matrices of single-precision floating point

numbers, ranging from 16 x 16 to 8192 x 8192 Time to move data from host to device and back is included in GPU

timing Conducted five rounds of tests and averaged the results

9

Small Problem Size Depending on the hardware configuration, the overhead

to copy the data may overwhelm the performance improvement of the GPU

10

Medium Problem Size GPU advantage becomes apparent as the matrix

size increases

11

Large Problem Size The GPUs can still finish a job in a matter of

seconds that takes several hours on the CPU

12

Speedup Summary

13

CUDA Computing System

A CUDA computing system consists of a host (CPU) and one or more devices (GPUs)

The portions of the program that can be evaluated in parallel are executed on the device. The host handles the serial portions and the transfer of execution and data to and from the device

14

CUDA Program Source Code

A CUDA program is a unified source code encompassing both host and device code. Convention: program_name.cu

NVIDIA’s compiler (nvcc) separates the host and device code at compilation

The host code is compiled by the host’s standard C compilers. The device code is further compiled by nvcc for execution on the GPU

15

CUDA Program Execution

Execution of a CUDA program begins on the host CPU When a kernel function (or simply “kernel”) is

launched, execution is transferred to the device and a massive “grid” of lightweight threads is spawned

When all threads of a kernel have finished executing, the grid terminates and control of the program returns to the host until another kernel is launched

16

CUDA Program Execution

17

TIME

CUDA Program Structure exampleint main(void) { float *a_h, *a_d; // pointers to host and device arrays const int N = 10; // number of elements in array size_t size = N * sizeof(float); // size of array in memory // allocate memory on host and device for the array // initialize array on host (a_h) // copy array a_h to allocated device memory location (a_d)

// kernel invocation code – to have the device perform // the parallel operations

// copy a_d from the device memory back to a_h // free allocated memory on device and host}

18

Data Movement and Memory Management In CUDA, host and device have separate memory spaces To execute a kernel, the program must allocate memory

on the device and transfer data from the host to the device

After kernel execution, the program needs to transfer the resultant data back to the host memory and free the device memory

C functions: malloc(), free()CUDA functions: cudaMalloc(), cudaMemcpy(), and cudaFree()

19

Data Movement exampleint main(void){ float *a_h, *a_d; const int N = 10; size_t size = N * sizeof(float); // size of array in memory

a_h = (float *)malloc(size); // allocate array on host cudaMalloc((void **) &a_d, size); // allocate array on device for (i=0; i<N; i++) a_h[i] = (float)i; // initialize array cudaMemcpy(a_d, a_h, size, cudaMemcpyHostToDevice);

// kernel invocation code

cudaMemcpy(a_h, a_d, sizeof(float)*N, cudaMemcpyDeviceToHost); cudaFree(a_d); free(a_h); // free allocated memory}

20

Execution Parameters and Kernel Launch

A kernel is invoked by the host program with execution parameters surrounded by ‘<<<’ and ‘>>>’ as in:

function_name <<< grid_dim, block_dim >>> (arg1, arg2);

At kernel launch, a “grid” is spawned on the device. A grid consists of a one- or two-dimensional array of “blocks”. In turn, a block consists of a one-, two-, or three-dimensional array of “threads”

Grid and block dimensions are passed to the kernel function at invocation as execution parameters

21

Execution Parameters and Kernel Launch gridDim and blockDim are CUDA built-in variables of

type dim3, essentially a C struct with three unsigned integer fields, x, y, and z

Since a grid is generally two-dimensional, gridDim.z is ignored but should be set to 1 for clarity

dim3 grid_d = (n_blocks, 1, 1); // this is stilldim3 block_d = (block_size, 1, 1); // host codefunction_name <<< grid_d, block_d >>> (arg1, arg2);

For one-dimensional grids and blocks, scalar values can be used instead of dim3 type

22

Execution Parameters and Kernel Launch

23

dim3 grid_dim = (2, 2, 1)dim3 block_dim = (4, 2, 2)

Limits on gridDim and blockDim The maximum size of a block (blockDim.x *

blockDim.y * blockDim.z) is 512 threads, regardless of dimension. You cannot increase the number of allowed threads by adding another dimension

Since a block is limited to 512 threads, one block per grid will usually not be sufficient

The values of gridDim.x and gridDim.y can range from 1 to 65,535

24

Kernel Invocation exampleint main(void){ float *a_h, *a_d; const int N = 10; size_t size = N * sizeof(float); // size of array in memory

a_h = (float *)malloc(size); // allocate array on host cudaMalloc((void **) &a_d, size); // allocate array on device for (i=0; i<N; i++) a_h[i] = (float)i; // initialize array cudaMemcpy(a_d, a_h, size, cudaMemcpyHostToDevice);

int block_size = 4; // set up execution parameters int n_blocks = N/block_size + (N%block_size == 0 ? 0:1); square_array <<< n_blocks, block_size >>> (a_d, N);

cudaMemcpy(a_h, a_d, sizeof(float)*N, cudaMemcpyDeviceToHost); cudaFree(a_d); free(a_h); // free allocated memory}

25

Kernel Functions A kernel function specifies the code to be executed by all

threads in parallel – an instance of single-program, multiple-data (SPMD) parallel programming.

A kernel function declaration is a C function extended with one of three keywords: “__device__”, “__global__”, or “__host__”.

26

Executed on the:

Only callable from the:

__device__ float DeviceFunc()

device device

__global__ void KernelFunc()

device host

__host__ float HostFunc() host host

CUDA Thread Organization Since all threads of a grid execute the same code,

they rely on a two-level hierarchy of coordinates to distinguish themselves: blockIdx and threadIdx

The example code fragment:ID = blockIdx.x * blockDim.x + threadIdx.x;will yield a unique ID for every thread across all blocks of a grid

27

Kernel Function and Threading example

CUDA kernel function:__global__ void square_array(float *a, int N) { int idx = blockIdx.x * blockDim.x + threadIdx.x; if (idx<N) a[idx] = a[idx] * a[idx];}

Compare with serial C version:void square_array(float *a, int N) { int i; for (i = 0; i < N; i++) a[i] = a[i] * a[i];}

28

CUDA Device Memory Types

Global Memory and Constant Memory can be accessed by the host and device. Constant Memory serves read-only data to the device at high bandwith. Global Memory is read-write and has a longer latency

Registers, Local Memory, and Shared Memory are accessable only to the device. Registers and Local Memory are available only to their own thread. Shared Memory is accessable to all threads within the same block

29

CUDA Memory Model

30

Host

CUDA Variable Type Qualifiers

31

Variable Declaration Memory Scope Lifetime

Non-Array Automatic Variables Register Thread Kernel

Automatic Array Variables Local Thread Kernel

__device__, __shared__, int var1 Shared Block Kernel

__device__, int var2 Global Grid Application

__device__, __constant__, int var3 Constant Grid Application

Sychronization Threads in the same block can synchronize using

__syncthreads()

When used in an if-then-else construction, all threads must branch on the same path or they will wait on each other forever, i.e., the __syncthreads() in the if branch is distinct from the __syncthreads() in the else branch

Threads in different blocks cannot perform barrier synchronization with each other. This is a major constraint, but it comes as part of a a big scalability trade-off

32

Transparent Scalability CUDA’s synchronization constraint allows blocks to be

executed in any order relative to each other, providing transparent scalability

The exact same code can be executed on devices with different execution resources. The execution time is inversely proportional to the available resources

33

CUDA Atomic Operations Race condition review: When two or more

concurrently running threads access a shared data item and the result depends on the order of execution.

We use “Atomics” to solve this problem

34

Deposit Operation:

load balanceadd amountstore balance

Desired Action:Balance: 100Deposit: 10Deposit: 200Balance: 310

Possible Problem:Balance: 100load 100add 10load 100add 200store 110store 300Balance: 300

CUDA Atomic Operations Race conditions are exceptionally problematic in

massively parallel programs when thousands of threads access data simultaneously

CUDA provides many atomic functions for integers including atomicAdd(), atomicSub(), atomicExch(), atomicMin(), and atomicMax()

Atomic operations can create bottlenecks which collapse your parallel program to a serial program and significantly degrade performance

Use sparingly and use wisely

35

Example of using a CUDA atomic wisely to find a global maximum

Naïve approach:__global__ void global_max(int* values, int* gl_max) {

int i = threadIdx.x + blockDim.x * blockIdx.x; int val = values[i]; atomicMax(gl_max,val);

}

Better idea:__global__ void global_max(int* values, int* max, int* regional_maxes, int num_regions) {

// int i and val as beforeif(atomicMax(&reg_max[region],val) < val) {

atomicMax(max,val); }

}

36

Example of using a CUDA atomic wisely to find a global maximum

Naïve approach:__global__ void global_max(int* values, int* gl_max) {

int i = threadIdx.x + blockDim.x * blockIdx.x; int val = values[i]; atomicMax(gl_max,val);

}

Better idea:__global__ void global_max(int* values, int* max, int* regional_maxes, int num_regions) {

// int i and val as beforeif(atomicMax(&reg_max[region],val) < val) {

atomicMax(max,val); }

}

37

Georgia Advanced Computing Resource Center (GACRC)

GACRC resources Requesting an account Setting up the user environment (i.e.: path

variables, etc.) Compiling a CUDA program using nvcc Creating a submission shell script Submitting a job to the queue

38

My First CUDA Program squares.c and squares.cu are identical serial C

programs Edit squares.cu with CUDA keywords to port to

parallel program (leave squares.c clean to refer back to if necessary)

Compile with makefile Create submission shell script Submit to queue

39

More CUDA Training Resources University of Georgia CUDA Teaching Center: http://cuda.uga.edu Nvidia training and education site: http://developer.nvidia.com/cuda-

education-training Stanford University course on iTunes U: http://itunes.apple.com/us/itunes-

u/programming-massively-parallel/id384233322 University of Illinois: http://courses.engr.illinois.edu/ece498/al/Syllabus.html University of California, Davis:

https://smartsite.ucdavis.edu/xsl-portal/site/1707812c-4009-4d91-a80e-271bde5c8fac/page/de40f2cc-40d9-4b0f-a2d3-e8518bd0266a

University of Wisconsin: http://sbel.wisc.edu/Courses/ME964/2011/me964Spring2011.pdf

University of North Carolina at Charlotte: http://coitweb.uncc.edu/~abw/ITCS6010S11/index.html

40

References Kirk, D., & Hwu, W. (2010). Programming Massively Parallel Processors: A

Hands-on Approach, 1 – 75 Tarjan, D. (2010). Introduction to CUDA, Stanford University on iTunes U Atallah, M. J. (Ed.), (1998). Algorithms and theory of computation handbook.

Boca Raton, FL: CRC Press von Neumann, J. (1945). First draft of a report on the EDVAC. Contract No.

W-670-ORD-4926, U.S. Army Ordnance Department and University of Pennsylvania

Sutter, H., & Larus, J. (2005). Software and the concurrency revolution. ACM Queue, 3(7), 54 – 62

Stratton, J. A., Stone, S. S., & Hwu, W. W. (2008). MCUDA: And efficient implementation of CUDA kernels for multi-core CPUs. Canada: Edmonton

Vandenbout, Dave (2008). My First Cuda Program, http://llpanorama.wordpress.com/2008/05/21/my-first-cuda-program/

41