AMS 148 Chapter 8: Optimization in CUDA, and Advanced Topics
Steven Reeves
1
Optimizing Data Transfers in CUDA C/C++
This section we will discuss code optimization with how to eﬃciently transfer data between the host and the
device. The peak bandwidth between the device memory and the GPU is much higher (720 GB/s for the
Tesla P100 found in Hummingbird) than the peak bandwidth between host memory and device memory (8
GB/s on a PCIex16 Generation 2). This disparity means that if your implementation requires many data
transfers from GPU to host or vice-versa it will greatly halter your performance. Let us begin with a few
general guidlines for host-device data transfers. We wish to:
• Minimize the ammount of data transferred between host and device when possible, even if that means
running kernels on the GPU that get little or no speed-up compared to running them on the host CPU.
• Hihger bandwidth is possible between the host and the device when using page-locked (or ’pinned’)
memory.
• Batching many small transfers into one larger transfer performs better as it eliminates most of the
per-transfer overhead.
• Data transfers between the host and device can sometimes be overlapped with kernel execution and
other data transfers.
First let us talk about how to measure time spent in data transfers without modifying source code.
1.1
Measuring Data Transfers Times with nvprof
To measure the time spent during each data transfer, we could record a CUDA event before and after each
transfer and use cudaEventElaspsedTime() as we have in the past. However, we can retrieve the elapsed
transfer time without deploying CUDA events by using nvprof, a command-line CUDA proﬁler included
with the CUDA Toolkit. Let’s try the following code example:
Listing 1: Example Code for Proﬁling
int
main ()
{
const
unsigned
int N = 1048576;
const
unsigned
int
bytes = N*sizeof(int);
int *h_a = (int*) malloc(bytes);
int *d_a;
cudaMalloc (( int **)&d_a , bytes);
memset(h_a , 0, bytes);
cudaMemcpy(d_a , h_a , bytes , cudaMemcpyHostToDevice );
cudaMemcpy(h_a , d_a , bytes , cudaMemcpyDeviceToHost );
return
0;
}
To proﬁle this code, we just need to compile it using nvcc, and the run nvprof with the program ﬁleneame
as an argument.
Listing 2: Running nvprof
$ nvcc
profile.cu -o profile.exe
$ nvprof ./ profile.exe
1
Using the Citrisdance (NVIDIA K20) server I recieve the following output:
==17821==
NVPROF
is
profiling
process
17821 ,
command: ./ profile.exe
==17821==
Profiling
application : ./ profile.exe
==17821==
Profiling
result:
Type
Time (%)
Time
Calls
Avg
Min
Max
Name
GPU
activities:
51.35%
1.5589 ms
1
1.5589 ms
1.5589 ms
1.5589 ms
[CUDA
memcpy
DtoH]
48.65%
1.4772 ms
1
1.4772 ms
1.4772 ms
1.4772 ms
[CUDA
memcpy
HtoD]
API
calls:
98.80%
480.32 ms
1
480.32 ms
480.32 ms
480.32 ms
cudaMalloc
0.93%
4.5109 ms
2
2.2554 ms
1.7877 ms
2.7232 ms
cudaMemcpy
0.21%
1.0257 ms
188
5.4550 us
220 ns
205.44 us
cuDeviceGetAttribute
0.04%
197.15 us
2
98.574 us
66.692 us
130.46 us
cuDeviceTotalMem
0.02%
93.452 us
2
46.726 us
44.398 us
49.054 us
cuDeviceGetName
0.00%
6.8620 us
4
1.7150 us
342 ns
5.2240 us
cuDeviceGet
0.00%
3.6260 us
3
1.2080 us
347 ns
2.4570 us
cuDeviceGetCount
We see that nvprof gives a full breakdown of the program, including the CUDA API calls and the GPU
activities. We see that the majority of the time is used on the memory allocation, but barring this API call,
the next most time consuming are the memory transfers. Memory transfers are much more common than
allocation in most applications.
1.2
Minimizing Data Transfers
We should not use only the GPU exectuation time of a kernel relative to CPU functions to decided whether
to run the GPU or CPU version. We also need to consider the cost of moving data across the PCI-e bus,
especially when we are initially porting code to CUDA. Because of the hetrogenous programming model of
CUDA (using both the CPU and GPU), code can be ported to the GPU one kernel at a time. In the initial
stages of writing CUDA code, data transfers may donimate the overall execution time. It is worth while to
monitor time spent on data transfer separately from time spent during computation within a kernel. It is
easy to use the command-line proﬁler for this, as demonstrated above. As more of our code is ported to
CUDA, we will remove intermediate transfers and descrease the overall execution time correspondingly.
1.3
Pinned Memory
Pageable memory space means memory contents that can be paged in/ paged out between DRAM and a
secondary storage device. Host memory allocations are generally pageable.
The main motivation for using pinned memory is to perform asynchronous transfers of data from the host
to the device. This is accomplished by using the CUDA primitive cudaMemcpyAsync and related functions.
Additionally, certain performance beneﬁts come from using pinned (or page-locked memory). In this section
we will give a few examples about how to allocate pinned memory and we investigate features of this type
of memory.
1.3.1
About pinned memory
The workings of paged memory is best described by the post on http://devblogs.nvidia.com which will be
paraphrased here.
Data allocations on the CPU are pageable by default. The GPU cannot access data directly from pageable
host memory, so when a data transfer from pageable host memory to device is invoked, the CUDA driver
will ﬁrst allocate a temporary page-locked, or ”pinned” host array. Then the data is copied into this pinned
array for transfer to the device. An illustration of this process is provided by Nvidia blogpost:
2
Figure 1:
Regular Data Transfer vs Pinned Data Transfer; NVIDIA Developer Blog
As shown in ﬁgure 1, pinned memory is used as a staging area for transfers from the device to the hsot.
We can avoid the cost of the transfer betweeen pageable and pinned host arrays by directly allocation the
host arrays in pinned memory. Allocate pinned host memory in CUDA C/C++ using cudaMallocHost() or
cudaHostAlloc() and free the memory with cudaFreeHost(). It is possible for pinned memory allocation
to fail, so we should check for errors using the cudaError t class. The following code demonstrates allocation
of pinned memory with error chekcing.
cudaError_t
status = cudaMallocHost (( void **)&h_aPinned , bytes);
if (status != cudaSuccess)
printf("Error
allocating
pinned
host
memory\n");
Data transfers using host pinned memory use the same cudaMemcpy() syntax as transfers with pageable
memory. We can use the following ”bandwidthtest” program to compare the rates between pagable and
pinned transfer rates.
Listing 3: Bandwidth Test
#include
<stdio.h>
#include
<assert.h>
//
Convenience
function
for
hcekcing
CUDA
runtime
API
results
// can be
wrapped
around
any
runtime
API
call. No -op in
release
builds.
inline
cudaError_t
checkCuda(cudaError_t
result)
{
#if
deifned(DEBUG) ||
defined(_DEBUG)
if( result !=
cudaSuccess ) {
fprintf(stderr , "CUDA
Runtime
Error: %s\n",
cudaGetErrorString (result));
assert(result ==
cudaSuccess );
}
#endif
return
result;
}
void
profileCopies (float
*h_a ,
float
*h_b ,
float
*d,
unsigned
int
n,
char
*desc)
{
printf("\n%s transfers\n", desc);
unsigned
int
bytes = n * sizeof(float);
// events
for
timing
cudaEvent_t
startEvent , stopEvent;
checkCuda( cudaEventCreate (& startEvent) );
checkCuda( cudaEventCreate (& stopEvent) );
checkCuda( cudaEventRecord (startEvent , 0) );
checkCuda( cudaMemcpy(d, h_a , bytes , cudaMemcpyHostToDevice ) );
checkCuda( cudaEventRecord (stopEvent , 0) );
checkCuda( cudaEventSynchronize (stopEvent) );
3
float
time;
checkCuda( cudaEventElapsedTime (&time , startEvent , stopEvent) );
printf("
Host to Device
bandwidth (GB/s): %f\n", bytes * 1e-6 / time);
checkCuda( cudaEventRecord (startEvent , 0) );
checkCuda( cudaMemcpy(h_b , d, bytes , cudaMemcpyDeviceToHost ) );
checkCuda( cudaEventRecord (stopEvent , 0) );
checkCuda( cudaEventSynchronize (stopEvent) );
checkCuda( cudaEventElapsedTime (&time , startEvent , stopEvent) );
printf("
Device to Host
bandwidth (GB/s): %f\n", bytes * 1e-6 / time);
for (int i = 0; i < n; ++i) {
if (h_a[i] != h_b[i]) {
printf("*** %s transfers
failed
***\n", desc);
break;
}
}
// clean up
events
checkCuda( cudaEventDestroy (startEvent) );
checkCuda( cudaEventDestroy (stopEvent) );
}
int
main ()
{
unsigned
int
nElements = 4*1024*1024;
const
unsigned
int
bytes = nElements * sizeof(float);
// host
arrays
float *h_aPageable , *h_bPageable;
float *h_aPinned , *h_bPinned;
// device
array
float *d_a;
//
allocate
and
initialize
h_aPageable = (float *) malloc(bytes);
// host
pageable
h_bPageable = (float *) malloc(bytes);
// host
pageable
checkCuda( cudaMallocHost (( void **)&h_aPinned , bytes) ); // host
pinned
checkCuda( cudaMallocHost (( void **)&h_bPinned , bytes) ); // host
pinned
checkCuda( cudaMalloc (( void **)&d_a , bytes) );
//
device
for (int i = 0; i < nElements; ++i) h_aPageable [i] = i;
memcpy(h_aPinned , h_aPageable , bytes);
memset(h_bPageable , 0, bytes);
memset(h_bPinned , 0, bytes);
// output
device
info
and
transfer
size
cudaDeviceProp
prop;
checkCuda( cudaGetDeviceProperties (&prop , 0) );
printf("\nDevice: %s\n", prop.name);
printf("Transfer
size (MB): %d\n", bytes / (1024 * 1024));
//
perform
copies
and
report
bandwidth
profileCopies (h_aPageable , h_bPageable , d_a , nElements , "Pageable");
profileCopies (h_aPinned , h_bPinned , d_a , nElements , "Pinned");
printf("n");
//
cleanup
cudaFree(d_a);
cudaFreeHost (h_aPinned);
cudaFreeHost (h_bPinned);
free( h_aPageable );
free( h_bPageable );
return
0;
}
The data transfer rate can depend on the type of host stysem (motherboard, CPU, and chipset) as well
as the GPU. Running this program on HummingBird we have the following output
Listing 4: Output of Bandwidth Test
Device: Tesla P100 -PCIE -16GB
Transfer
size (MB): 16
4
Pageable
transfers
Host to Device
bandwidth (GB/s): 2.990821
Device
to Host
bandwidth (GB/s): 4.364375
Pinned
transfers
Host to Device
bandwidth (GB/s): 12.017788
Device
to Host
bandwidth (GB/s): 12.726673
On Hummingbird the CPU is an AMD Opteron processor and sets a decent pageable transfer rate.
However, we ﬁnd that the pinned transfer rate is much more impressive, oﬀering more than 4 times bandwidth
Host to Device and 3 times bandwidth Device to Host. Note that the pageable transfer rate depends on the
speed of the CPU. Faster CPUs will oﬀer a better pageable bandwidths, however with modern GPUs pinned
bandwidths will exceed this capability.
A word of warning, you should not over-allocate pinned memory. Doing so can reduce overall system
performance because it reduces the amount of physical memory available to the operating system and other
programs. How much is too much is diﬃcult to tell a priory, so as with most optimizaitons, test your code
and the systems they run on for optimal performance parameters.
1.3.2
Batching Small Transfers
Due to the overhead associated with CPU to GPU memory transfers, it is better to batch many small
transfers together into a single transfer. This is easy to do by using a temporary array, preferable pinned,
and packing it with the data to be transferred.
2
CUDA Streams
In the previous section we discussed how to transfer data eﬃciently between the host and device. In this
section, we discuss how to overlap data transfers with computation on the host, computation on the device,
and other data transfers between the hsot and device. Through the use of CUDA streams, overlap between
data transfers and other operations can be achieved.
A stream in CUDA is a sequence of operations that execute on the device in the order in which they
are issued by the host code. While operations within a stream are garaunteed to execute in the prescribed
order, operations in diﬀerent streams can be interleaved and, when possible, they will run concurrently.
2.1
The default stream
All device operations (kernels and data transfers) in CUDA run in a stream. When no stream is speciﬁed,
the default stream (also referred to the ”null stream”) is used. The default stream is diﬀerent from other
streams as it is a synchronizing stream with respect to the device: no operation on the default stream will
begin until all previous issued operation in any stream on the device have completed, and an operation in
the default stream must complete before any other operation (in any stream on the device) will begin.
Let us check out a simple code snippet that use the default stream:
Listing 5: Code Snippet Default Stream
cudaMemcpy(d_a , a, numBytes , cudaMemcpyHostToDevice );
increment <<<1,N>>>(d_a)
cudaMemcpy(a, d_a , numBytes , cudaMemcpyDeviceToHost );
In this snippet, from the persepctive of the PGU¡ all three operation are issued to the same (default) stream
and will execute in the order that they were issued.
From the perspective of the CPU, the implicit data transfers are blocking or synchronous transfers, while
kernel launching is asynchronous. Since the host-to-device transfer on the ﬁrst line is synchronous, the CPU
thread will not reach the kernel call on the second line until the host to device transfer is complete. Once
the kernel is issued, the CPU thread moves to the third line, but the transfer on that line will not begin due
to the device-side order of execution.
The asynchronous behavior of the kernel launches from the CPU’s perspective makes overlapping device
and host computations very easy, take the following snippet for example:
5
cudaMemcpy(d_a , a, numBytes , cudaMemcpyHostToDevice );
increment <<<1,N>>>(d_a)
myCpuFunction (b)
cudaMemcpy(a, d_a , numBytes , cudaMemcpyDeviceToHost );
In this code, as soon as the kernel is launched on the device the CPU thread executes myCpuFunction(),
overlapping the CPU function with the kernel execution on the GPU. Whether the host function or device
kernel completes ﬁrst doesn’t aﬀtect the subsequent device to host memory transfer, which begins only after
the kernel is completed. From the perspective of the GPU, nothign has changed from the previous example;
the device is completely unaware of myCpuFunction().
2.2
Non-default streams
Streams other than the null CUDA C/C++ are declared, created, and destroyed in the host code, as in this
example:
Listing 6: Streams
cudaStream_t
stream1;
cudaError_t
result;
result = cudaStreamCreate (& stream1);
result = cudaStreamDestroy (& stream1);
In order to issue a data transfer to a non default CUDA stream, we use the cudaMemcpyAsync() function,
which is similar to the regular cudaMemcpy() function we have been using before, but it takes another
argument.
result = cudaMemcpyAsyn (d_a , a, N, cudaMemcpyHostToDevice , stream1);
This function is asynchronous with the host, so control returns to the host thread immediately after the
transfer is issued. There are 2D and 3D extensions of this as well. To issue a kernel to a non-default stream,
we specify the stream in question as a fourth argument in the kernel conﬁguration:
increment <<<1,N,0,stream1 >>>(d_a);
2.3
Synchronization with streams
Since all operations in non-default streams are asynchronous with respect to the host code, you will ﬁnd
that there may be situations that will require you to synchronize all streams. There are several ways to do
this. The brute force way to do this is to utilize cudaDeviceSynchronize() which creates a barrier on the
host thread until all previously issued GPU operations are complete. In most cases this is too much, and
can hurt performance due to stalling the device and host thread.
The CUDA stream API has multiple less severe methods of synchronization.
The function
cudaStreamSynchronize(stream) can be used to block the host from prorceeding until all previous is-
sued operations on the speciﬁed stream have completed. The function cudaStreamQuery(stream) tests
whether all operations issued to the speciﬁed stream have completed, without blocking host execution. The
fuctions cudaEventSyncrhonize(event) and cudaEventQuery(event) do the similar only that their result
is based oﬀof whether a particular event has been recorded. You can also synchronize operations within a
single stream on a speciﬁc event using cudaStreamWaitEvent(event).
2.4
Overlapping Kernel Execution and Data Transfer
Previously we demonstrated overlapping kernel execution in the default stream with execution of operations
on the CPU. But our goal in this section is to show how to overlap kernel execution with data transfers.
There are several requirements for this to happen:
• The device must be capable of ”concurrent copy and exectuion”
• The kernel execution and the data transfer to be overlapped must both occur in diﬀerent, non default
streams
• The host memory involved in the data transfer must be pinned memory.
6
So we’re going to modify the simple code from the previous section, the full code available on github. In this
modiﬁed snippet, we break up the array of size N into chunks of streamSize elements. SInce the kernel
operates independently on all elements, each of the chunks can be processed indepednently. The number of
(non-default) streams used is nStreams=N/streamSize. There are multiple ways to implement the domain
decompostion of the data and processing, one way is to loop over the operations for each chunk.
for (int i = 0; i < nStreams; ++i) {
int
offset = i * streamSize;
cudaMemcpyAsync (& d_a[offset], &a[offset], streamBytes , cudaMemcpyHostToDevice , stream[i]);
kernel <<<streamSize/blockSize , blockSize , 0, stream[i]>>>(d_a , offset);
cudaMemcpyAsync (&a[offset], &d_a[offset], streamBytes , cudaMemcpyDeviceToHost , stream[i]);
}
Another approach is to batch similar operations together, issuing all the host to device transfers ﬁrst, followed
by the kernel lauches, lastly the device to host transfers.
for (int i = 0; i < nStreams; ++i) {
int
offset = i * streamSize;
cudaMemcpyAsync (& d_a[offset], &a[offset],
streamBytes , cudaMemcpyHostToDevice , cudaMemcpyHostToDevice , stream[i]);
}
for (int i = 0; i < nStreams; ++i) {
int
offset = i * streamSize;
kernel <<<streamSize/blockSize , blockSize , 0, stream[i]>>>(d_a , offset);
}
for (int i = 0; i < nStreams; ++i) {
int
offset = i * streamSize;
cudaMemcpyAsync (&a[offset], &d_a[offset],
streamBytes , cudaMemcpyDeviceToHost , cudaMemcpyDeviceToHost , stream[i]);
}
Both asyncrhonous methods shown above yield the correct results, and in both cases dependent operations
are issued to the same stream in the order in which they need to be executed. The results can vary depending
on GPU architectures: On citrisdance a Kepler based server I get the following results:
Device : Tesla
K20c
Time
for
sequential
transfer
and
execute (ms): 7.590112
max
error: 1.192093e -07
Time
for
asynchronous
V1
transfer
and
execute (ms): 3.995456
max
error: 1.192093e -07
Time
for
asynchronous
V2
transfer
and
execute (ms): 3.975712
max
error: 1.192093e -07
However, on Hummingbird the performance is faster:
Device : Tesla P100 -PCIE -16 GB
Time
for
sequential
transfer
and
execute (ms): 3.154144
max
error: 1.192093e -07
Time
for
asynchronous
V1
transfer
and
execute (ms): 1.971200
max
error: 1.192093e -07
Time
for
asynchronous
V2
transfer
and
execute (ms): 1.959584
max
error: 1.192093e -07
This is mostly due to hardware advances in the years, but also due to changes in compute combatibility.
For devices that have compute cability 3.5 or higher, the Hyper-Q feature eliminates the need to tailor the
launch order so either approach yields similar results. However, on older devices, such as the Tesla C1060,
a compute 1.5 device the results are diﬀerent.
Device : Tesla
C1060
Time
for
sequential
transfer
and
execute (ms ): 12.92381
max
error : 2.3841858E
-07
Time
for
asynchronous
V1
transfer
and
execute (ms ): 13.63690
max
error : 2.3841858E
-07
Time
for
asynchronous
V2
transfer
and
execute (ms ): 8.84588
max
error : 2.3841858E
-07
We see that with this device the version one transfer runs slower than the serially transfered method. A good
way to understand this is that the C1060 only has one copy engine and one kernel engine. The following
diagram illustrates the method of operation on the C1060
7
Figure 2: Singe Engine, Nvidia Developer Blogs
whereas a newer device, say a C2050, which contains two copy engines and kernel engines, the timelines
are more like this:
8
Figure 3: Multiple Engines, Nvidia Developer Blogs
The number of engines in these previous models dictated the nature of asynchronous operations. The
Hyper-Q ﬁrmware allows for more eﬀective use of the grid management. The following illustrations show
the proﬁling of an applicaiton without and with hyper-Q.
Figure 4: Proﬁling of older multi-engine GPUs without Hyper-Q
9
Figure 5: Multi-engine GPUs with Hyper-Q
GPUs with Hyper-Q allow the hardward to compact asynchronous launches using either method. Allow-
ing the developer to not worry (as much) about the hardware implementation.
3
Optimizing Calculations
In the previous sections we looked into to optimizing memory transactions, in this section we’ll look into
optimizing calculations within kernels.
As an example we will follow Mark Harris’ Optimizing Parallel
Reduction in CUDA presentation.
Using this, we will discuss 7 diﬀerent versions of the reduction kernel.
Using these versions we will
look at several important optimization strategies. Utilizing these strategies we strive to reach GPU peak
performance. We will need to choose the right metric, FLOP/s for compute-bound kernels, and Bandwidth
for memory-bound kernels. Reductions have very low arithmetic intensity, 1 ﬂop per element loaded, making
this bandwidth optimal. The following code is tested on aa Nvidia Titan XP GPU, which has a theoretical
peak bandwidth of 547.6 GB/s. In what follows we will try to design an algorithm that gets close to this
theoretical bandwidth. A true measurement is the eﬃctive bandwdith
BWeffective = (RB + WB)/(t × 109)
where RB is the number of bytes read by the kernel, and WB is the number of bytes written. We multiply
t by 109 to retrieve GB/s.
The reduction that we have had before utilizes a method called address interleaving.
Listing 7: First Reduction
template
<class T>
__global__
void
reduce0(T *g_idata , T *g_odata){
extern
__shared__ T sdata [];
// each
thread
loads
one
element
from
global to
shared
mem
unsigned
int tid = threadIdx.x;
unsigned
int i = blockIdx.x*blockDim.x + threadIdx.x;
sdata[tid] = g_idata[i];
__syncthreads ();
// do
reduction
in
shared
mem
for(unsigned
int s=1; s < blockDim.x; s *= 2) {
if (tid % (2*s) == 0)
{
sdata[tid] += sdata[tid + s];
}
__syncthreads ();
}
// write
result
for
this
block to
global
mem
if (tid == 0)
g_odata[blockIdx.x] = sdata [0];
}
The if statement within this code is highly divergent, it branches the threads and many launched threads
are not active, and can result in very poor performance. We will test this code and the code that follows
using 1 million integers.
10
Upon execution of this ﬁrst reduce, the elapsed time is 0.132096 ms, with a eﬀective bandwidth of 31.814
GB/s. This is far from the the theoretical bandwidth. Let us change this kernel to remove the divergent
branching.
Listing 8: Reduction without divergent branching
template
<class T>
__global__
void
reduce1(T *d_out , const T *d_in)
{
// sdata is
allocated
in the
kernel
call: via
dynamic
shared
memeory
extern
__shared__ T sdata [];
int
myId = threadIdx.x + blockDim.x*blockIdx.x;
int tid = threadIdx.x;
// load
shared
mem
from
global
mem
sdata[tid] = d_in[myId ];
__syncthreads (); // always
sync
before
using
sdata
//do
reduction
over
shared
memory
for(int s = 1; s<blockDim.x; s *=2)
{
int
index = 2*s*tid; // Strided
indexing!
if(index < blockDim.x)
{
sdata[index] += sdata[index + s];
}
__syncthreads (); // make
sure
all
additions
are
finished
}
// only
tid 0 writes
out
result!
if(tid == 0)
{
d_out[blockIdx.x] = sdata [0];
}
}
All we changed in this code was the divergent inner loop. In this case we moved to a strided indexing
scheme to create non-divergent branches. Upon execution of this kernel, we ﬁnd that the elapsed time is
now 0.071264 ms with an eﬀective bandwidth of 58.9709GB/s. This shows that we have a 1.85 times speed
up over the original. There’s an additional problem here, we have shared memory bank conﬂicts, induced
by the strided index.
We will change our looping scheme to yield sequential addressing, which will alleviate the shared memory
bank conﬂict. To do this we swap the inner loop to be the following
Listing 9: Reduce with sequential addressing
for(unsigned
int s = blockDim.x/2; s >0; s
>>=1)
{
if(tid < s)
{
sdata[tid] += sdata[tid+s];
}
__syncthreads (); // make
sure
all
additions
are
finished
}
all else the same. By executing this kernel, we have the following reports: Elapsed time : 0.062816ms,
Eﬀective Bandwidth : 66.9017 GB/s, yielding a 1.13× speed up from the previous kernel.
The problem with this implementation is that there are many idle threads, we only use half of the
threads in the thread block upon the ﬁrst loop of the iteration. In this case, we will half the number of
blocks used, and change the way we load into shared memeory. For this purpose we will do two loads, and
do the ﬁrst addition in the reduction kernel upon loading into shared memory. So, we change our kernel to
use:
Listing 10: First Add During Load
int
myId = threadIdx.x + (blockDim.x*2)*blockIdx.x;
int tid = threadIdx.x;
// load
shared
mem
from
global
mem
sdata[tid] = d_in[myId] + d_in[myId + blockDim.x];
__syncthreads (); // always
sync
before
using
sdata
11
Upon execution, we reduce the elapsed time to 0.034528ms and increase the bandwidth to 121.713GB/s.
Giving a speed up of 1.8192 times.
At 121GB/s we’re far from the bandwidth upper bound. Therefore there is likely a bottleneck in the
instruction overhead. Ancillary instructions that are not loads, stores, or arithmetic for core computation
can become bottlenecks. In this case, address arithmetic and loop overhead. Our strategy for mitigating
this will be to unroll loops.
In the next kernel we will unroll the last warp. That is, we will change our loop to end at s = 32. Note
that this will save useless work in every warp, since we will be taking stages out of the for loop. Without
the unroll, every warp executes all iterations of the for loop and if statement.
Listing 11: Unrolling the last warp
for(unsigned
int s = blockDim.x/2; s >32; s
>>=1)
{
if(tid < s)
{
sdata[tid] += sdata[tid+s];
}
__syncthreads (); // make
sure
all
additions
are
finished
}
if(tid < 32)
{
sdata[tid] += sdata[tid +32];
sdata[tid] += sdata[tid +16];
sdata[tid] += sdata[tid +8];
sdata[tid] += sdata[tid +4];
sdata[tid] += sdata[tid +2];
sdata[tid] += sdata[tid +1];
}
The eﬀects of unrolling the last warp is fairly noticable. We reduce the kernel time to 0.023936ms with
bandwidth being 175.572 GB/s, giving a speed up of 1.442513369 times from the last kernel.
Now if we know the know the number of iterations at compile time, then we could completely unroll the
reduction. Luckily, the block size is limiited by the GPU to 1024 threads, and we’re sticking to powers of
2 for blocksizes. So we can easily unroll for a ﬁxed block size, which we will use a precompiler derictive to
deﬁne blockSize. Using this value all if statements involving blockSize will be evaulated at compile time.
Therefore we can change the above for loop to a diﬀerent statment
Listing 12: Unrolling all the loops
if(blockSize
>= 512){
if(tid < 256)
{
sdata[tid] += sdata[tid + 256];
}
__syncthreads ();
}
if(blockSize
>= 256){
if(tid < 128)
{
sdata[tid] += sdata[tid + 128];
}
__syncthreads ();
}
if(blockSize
>= 128){
if(tid < 64)
{
sdata[tid] += sdata[tid + 64];
}
__syncthreads ();
}
if(tid < 32)
{
sdata[tid] += sdata[tid +32];
sdata[tid] += sdata[tid +16];
sdata[tid] += sdata[tid +8];
sdata[tid] += sdata[tid +4];
sdata[tid] += sdata[tid +2];
sdata[tid] += sdata[tid +1];
}
12
Using this method we shave the time down 0.02192 ms, and raise the bandwidth to 191.72GB/s, yielding a
moderate speed up just about 1.1.
Before getting to the last optimization, lets consider the complexity of the reduce algorithm. We know
that there are log2(N) parallel steps, where each step S does N/2S independent operations. For N = 2D,
the reduction algorithm performs
X
S∈[1···D]
2D−S = N −1
operations, making is work eﬃcient. With P threads operating in parallel, the time complexity of the reduce
is O(N/P + log2(N)). Now we need to think about cost, the cost of the parallel algorithm is the number of
processors times the time complexity. If we allocate N processors, then the cost is N log(N), which is not
cost eﬀecient. Brent’s theorem suggests that we use O(N/ log(N))threads. Each thread will do O(log(N))
sequential work. So all O(N/ log(N)) threads will cooperatie for O(log N) steps, therefore resulting in O(N)
cost. This is called algorithm cascading, and in practice can lead to signiﬁcant speed ups.
To cascade the reduction, we will combine sequential and parallel reduction methods. Each thread loads
and sums multiple elements into shared memory, then we will perform the tree based reduction in shared
memory. Brent’s theorem suggests that each thread should sum O(log(N)) elements. It can be sometimes
beneﬁcial to push this further, we can hide latency with more work per thread. We will do this using 32
elements per thread. The changes in this last reduction are almost entirely in the load/serial reduce:
Listing 13: Algorithm Cascading
int
myId = threadIdx.x + (blockDim.x*2)*blockIdx.x;
int tid = threadIdx.x;
int
gridSize = blockDim.x*2* gridDim.x;
sdata[tid] = 0;
// load
shared
mem
from
global
mem
while(myId < n)
{
sdata[tid] += d_in[myId] + d_in[myId + blockDim.x];
myId +=
gridSize;
}
__syncthreads ();
Then upon kernel launch we have
reduce6 <<<dimGrid6 , dimBlock , 1024* sizeof(int) >>>(reduced , array , 32);
Using the algorithm cascading we reduce the elapsed time down to 0.008192 ms, oﬀering 513GB/s bandwidth,
and giving a 2.67578 times speed up.
Kernel
Time 220 integers
Bandwidth
Step Speed Up
Cummulative Speed Up
Kernel 1
0.132096 ms
31.814 GB/s
-
-
Kernel 2
0.071264 ms
58.8198 GB/s
1.853x
1.853x
Kernel 3
0.062816 ms
66.9017 GB/s
1.134x
2.102x
Kernel 4
0.034528 ms
131.713 GB/s
1.819x
3.826x
Kernel 5
0.023936 ms
175.572 GB/s
1.442x
5.519x
Kernel 6
0.021920 ms
191.720 GB/s
1.092x
6.026x
Kernel 7
0.008192 ms
513.001 GB/s
2.676x
16.13x
Table 1: Performance of Reduction via optimization
So putting it all together, we can achieve a speed up of over 16 times. Here is an interesting oberservation:
• Algorithmic Optimizations By changing the addressing and using algorithm cascading we achieved
10.23× speed up collectively.
• Coding Optimization By unrolling the loops we only achieved 1.58× speed up collectively.
So a good rule of thumb is to optimizing your algorithm ﬁrst, then optimize your code using unrolling of
loops.
13
In conclusion – to fully optimize CUDA code, you should understand CUDA performance characteristics.
Mostly, memory coalescing, divergent branching, bank conﬂicts, and latency hiding. Consider the algorithm
that you are programming, ascertain if it is a compute limited algorithm or bandwidth limited.
Using
parallel algorithm complexity theory, we found how to cascade the algorithm, allowing for quite a substantial
optimization. Identify bottlenecks in your algorithm, like we did with the memory and instruction overhead.
Finally, be sure to optimize the your algorithm, and then optimize your code.
14


Fundamental CUDA
Optimization
NVIDIA Corporation
Outline
Fermi/Kepler Architecture
Kernel optimizations
Launch configuration
Global memory throughput
Shared memory access
Instruction throughput / control flow

Optimization of CPU-GPU interaction
Maximizing PCIe throughput
Overlapping kernel execution with memory copies Most concepts in this presentation apply to any language or API on NVIDIA GPUs
Most concepts in this
presentation apply to
any language or API
on NVIDIA GPUs
512 Scalar Processor (SP) cores execute parallel
thread instructions

16 Streaming Multiprocessors (SMs)
 each contains
32 scalar processors
32 fp32 / int32 ops / clock,
16 fp64 ops / clock
4 Special Function Units (SFUs)
Shared register file (128KB)
48 KB / 16 KB Shared memory
16KB / 48 KB L1 data cache
20-Series Architecture (Fermi)
Kepler cc 3.5 SM (GK110)
“SMX” (enhanced SM)
192 SP units (“cores”)
64 DP units
LD/ST units
4 warp schedulers
Each warp scheduler is dual-
issue capable
K20: 13 SMX’s, 5GB
K20X: 14 SMX’s, 6GB
K40: 15 SMX’s, 12GB
Software
Hardware
Threads are executed by scalar processors
Thread
Scalar
Processor
Thread
Block
Multiprocessor
Thread blocks are executed on multiprocessors

Thread blocks do not migrate

Several concurrent thread blocks can reside on one
multiprocessor - limited by multiprocessor
resources (shared memory and register file)
...
Grid
Device
A kernel is launched as a grid of thread blocks
Execution Model
Thread
Block
Multiprocessor
32 Threads
32 Threads
32 Threads
...
Warps
A thread block consists of
32-thread warps

A warp is executed
physically in parallel
(SIMD) on a multiprocessor
=
Warps
Host
CPU
Chipset
DRAM
Device
DRAM

Global
Constant
Texture
Local
GPU
Multiprocessor
Registers
Shared Memory
Multiprocessor
Registers
Shared Memory
Multiprocessor
Registers
Shared Memory
Constant and Texture
Caches
L1 / L2 Cache
Memory Architecture
© NVIDIA Corporation 2011
Launch Configuration
Launch Configuration
Key to understanding:
Instructions are issued in order
A thread stalls when one of the operands isn’t ready:
Memory read by itself doesn’t stall execution
Latency is hidden by switching threads
GMEM latency: 400-800 cycles
Arithmetic latency: 18-22 cycles
How many threads/threadblocks to launch?
Conclusion:
Need enough threads to hide latency

Launch Configuration
Hiding arithmetic latency:
Need ~18 warps (576 threads)  per SM
Or, latency can also be hidden with independent instructions from the
same warp
For example, if instruction never depends on the output of preceding
instruction, then only 9 warps are needed, etc.
Maximizing global memory throughput:
Depends on the access pattern, and word size
Need enough memory transactions in flight to saturate the bus
Independent loads and stores from the same thread
Loads and stores from different threads
Larger word sizes can also help (float2 is twice the transactions of float, for
example)
Maximizing Memory Throughput
Increment of an array of 64M elements
Two accesses per thread (load then store)
The two accesses are dependent, so really 1 access per thread at a time
Tesla C2050, ECC on, theoretical bandwidth: ~120 GB/s
Several independent smaller
accesses have the same effect
as one larger one.
For example:
Four 32-bit  ~=  one 128-bit
Launch Configuration: Summary
Need enough total threads to keep GPU busy
Typically, you’d like 512+ threads per SM
More if processing one fp32 element per thread
Of course, exceptions exist
Threadblock configuration
Threads per block should be a multiple of warp size (32)
SM can concurrently execute up to 8 thread blocks
Really small thread blocks prevent achieving good occupancy
Really large thread blocks are less flexible
I generally use 128-256 threads/block, but use whatever is best for the application
For more details:
Vasily Volkov’s GTC2010 talk “Better Performance at Lower Occupancy”
(http://www.gputechconf.com/page/gtc-on-demand.html#session2238)
Global Memory
Throughput
Memory Hierarchy Review
Local storage
Each thread has own local storage
Mostly registers (managed by the compiler)
Shared memory / L1
Program configurable: 16KB shared / 48 KB L1   OR   48KB shared / 16KB L1
Shared memory is accessible by the threads in the same threadblock
Very low latency
Very high throughput: 1+ TB/s aggregate
L2
All accesses to global memory go through L2, including copies to/from CPU host
Global memory
Accessible by all threads as well as host (CPU)
High latency (400-800 cycles)
Throughput: up to 177 GB/s
Memory Hierarchy Review
L2
Global Memory
Registers
L1 SM-N
SM-N
SMEM
Registers
L1 SM-0
SM-0
SMEM
Registers
L1 SM-1
SM-1
SMEM
GMEM Operations
Two types of loads:
Caching
Default mode
Attempts to hit in L1, then L2, then GMEM
Load granularity is 128-byte line
Non-caching
Compile with –Xptxas –dlcm=cg option to nvcc
Attempts to hit in L2, then GMEM
– Do not hit in L1, invalidate the line if it’s in L1 already
Load granularity is 32-bytes
Stores:
Invalidate L1, write-back for L2
Load Operation
Memory operations are issued per warp (32 threads)
Just like all other instructions
Operation:
Threads in a warp provide memory addresses
Determine which lines/segments are needed
Request the needed lines/segments
Caching Load
Warp requests 32 aligned, consecutive 4-byte words
Addresses fall within 1 cache-line
Warp needs 128 bytes
128 bytes move across the bus on a miss
Bus utilization: 100%

...
addresses from a warp
96
192
128
160
224
288
256
32
64
352
320
384
448
416
Memory addresses
0
Non-caching Load
Warp requests 32 aligned, consecutive 4-byte words
Addresses fall within 4 segments
Warp needs 128 bytes
128 bytes move across the bus on a miss
Bus utilization: 100%

...
addresses from a warp
96
192
128
160
224
288
256
32
64
352
320
384
448
416
Memory addresses
0
Caching Load
Warp requests 32 aligned, permuted 4-byte words
Addresses fall within 1 cache-line
Warp needs 128 bytes
128 bytes move across the bus on a miss
Bus utilization: 100%

...
96
192
128
160
224
288
256
32
64
352
320
384
448
416
Memory addresses
addresses from a warp
0
Non-caching Load
Warp requests 32 aligned, permuted 4-byte words
Addresses fall within 4 segments
Warp needs 128 bytes
128 bytes move across the bus on a miss
Bus utilization: 100%

...
96
192
128
160
224
288
256
32
64
352
320
384
448
416
Memory addresses
addresses from a warp
0
Caching Load
Warp requests 32 misaligned, consecutive 4-byte words
Addresses fall within 2 cache-lines
Warp needs 128 bytes
256 bytes move across the bus on misses
Bus utilization: 50%

96
192
128
160
224
288
256
...
addresses from a warp
32
64
0
352
320
384
448
416
Memory addresses
Non-caching Load
Warp requests 32 misaligned, consecutive 4-byte words
Addresses fall within at most 5 segments
Warp needs 128 bytes
160 bytes move across the bus on misses
Bus utilization: at least 80%
Some misaligned patterns will fall within 4 segments, so 100% utilization
96
192
128
160
224
288
256
...
addresses from a warp
32
64
0
352
320
384
448
416
Memory addresses
Caching Load
All threads in a warp request the same 4-byte word
Addresses fall within a single cache-line
Warp needs 4 bytes
128 bytes move across the bus on a miss
Bus utilization: 3.125%

...
addresses from a warp
96
192
128
160
224
288
256
32
64
352
320
384
448
416
Memory addresses
0
Non-caching Load
All threads in a warp request the same 4-byte word
Addresses fall within a single segment
Warp needs 4 bytes
32 bytes move across the bus on a miss
Bus utilization: 12.5%

...
addresses from a warp
96
192
128
160
224
288
256
32
64
352
320
384
448
416
Memory addresses
0
Caching Load
Warp requests 32 scattered 4-byte words
Addresses fall within N cache-lines
Warp needs 128 bytes
N*128 bytes move across the bus on a miss
Bus utilization:  128 / (N*128)

...
addresses from a warp
96
192
128
160
224
288
256
32
64
352
320
384
448
416
Memory addresses
0
Non-caching Load
Warp requests 32 scattered 4-byte words
Addresses fall within N segments
Warp needs 128 bytes
N*32 bytes move across the bus on a miss
Bus utilization:  128 / (N*32)

...
addresses from a warp
96
192
128
160
224
288
256
32
64
352
320
384
448
416
Memory addresses
0
Impact of Address Alignment
Warps should access aligned regions for maximum memory throughput
L1 can help for misaligned loads if several warps are accessing a contiguous
region
ECC further significantly reduces misaligned store throughput

Experiment:
– Copy 16MB of floats
– 256 threads/block

Greatest throughput
drop:
– CA loads:
15%
– CG loads:
32%

GMEM Optimization Guidelines
Strive for perfect coalescing
Align starting address (may require padding)
A warp should access within a contiguous region
Have enough concurrent accesses to saturate the bus
Process several elements per thread
Multiple loads get pipelined
Indexing calculations can often be reused
Launch enough threads to maximize throughput
Latency is hidden by switching threads (warps)
Try L1 and caching configurations to see which one works best
Caching vs non-caching loads (compiler option)
16KB vs 48KB L1 (CUDA call)
Shared Memory
Shared Memory
Uses:
Inter-thread communication within a block
Cache data to reduce redundant global memory accesses
Use it to improve global memory access patterns
Organization:
32 banks, 4-byte wide banks
Successive 4-byte words belong to different banks
Performance:
4 bytes per bank per 2 clocks per multiprocessor
smem accesses are issued per 32 threads (warp)
serialization: if N threads of 32 access different 4-byte words in the same
bank, N accesses are executed serially
multicast: N threads access the same word in one fetch
Could be different bytes within the same word
Bank Addressing Examples
No Bank Conflicts
No Bank Conflicts
Bank 31
Bank 7
Bank 6
Bank 5
Bank 4
Bank 3
Bank 2
Bank 1
Bank 0
Thread 31
Thread 7
Thread 6
Thread 5
Thread 4
Thread 3
Thread 2
Thread 1
Thread 0
Bank 31
Bank 7
Bank 6
Bank 5
Bank 4
Bank 3
Bank 2
Bank 1
Bank 0
Thread 31
Thread 7
Thread 6
Thread 5
Thread 4
Thread 3
Thread 2
Thread 1
Thread 0
Bank Addressing Examples
2-way Bank Conflicts
8-way Bank Conflicts
Thread 31
Thread 30
Thread 29
Thread 28
Thread 4
Thread 3
Thread 2
Thread 1
Thread 0
Bank 31
Bank 7
Bank 6
Bank 5
Bank 4
Bank 3
Bank 2
Bank 1
Bank 0
Thread 31
Thread 7
Thread 6
Thread 5
Thread 4
Thread 3
Thread 2
Thread 1
Thread 0
Bank 9
Bank 8
Bank 31
Bank 7
Bank 2
Bank 1
Bank 0
x8
x8
Shared Memory: Avoiding Bank Conflicts
32x32 SMEM array
Warp accesses a column:
32-way bank conflicts (threads in a warp access the same bank)

31
2
1
0
31
2
1
0
31
2
1
0
warps:
0         1         2              31
Bank 0
Bank 1
  …
Bank 31
2
0
1
31
Shared Memory: Avoiding Bank Conflicts
Add a column for padding:
32x33 SMEM array
Warp accesses a column:
32 different banks, no bank conflicts

31
2
1
0
31
2
1
0
31
2
1
0
warps:
0         1         2             31       padding
Bank 0
Bank 1
  …
Bank 31
31
2
0
1
Instruction Throughput
& Control Flow
Runtime Math Library and Intrinsics
Two types of runtime math library functions
__func(): many map directly to hardware ISA
Fast but lower accuracy (see CUDA Programming Guide for full details)
Examples: __sinf(x), __expf(x), __powf(x, y)
func(): compile to multiple instructions
Slower but higher accuracy (5 ulp or less)
Examples: sin(x), exp(x), pow(x, y)

A number of additional intrinsics:
__sincosf(), __frcp_rz(), ...
Explicit IEEE rounding modes (rz,rn,ru,rd)
Control Flow
Instructions are issued per 32 threads (warp)
Divergent branches:
Threads within a single warp take different paths
if-else, ...
Different execution paths within a warp are serialized
Different warps can execute different code with no impact on
performance
Avoid diverging within a warp
Example with divergence:
if (threadIdx.x > 2) {...} else {...}
Branch granularity < warp size
Example without divergence:
if (threadIdx.x / WARP_SIZE > 2) {...} else {...}
Branch granularity is a whole multiple of warp size
CPU-GPU Interaction
Pinned (non-pageable) memory
Pinned memory enables:
faster PCIe copies
memcopies asynchronous with CPU
memcopies asynchronous with GPU
Usage
cudaHostAlloc / cudaFreeHost
instead of malloc / free
cudaHostRegister / cudaHostUnregister
pin regular memory after allocation
Implication:
pinned memory is essentially removed from host virtual memory

Streams and Async API
Default API:
Kernel launches are asynchronous with CPU
Memcopies (D2H, H2D) block CPU thread
CUDA calls are serialized by the driver
Streams and async functions provide:
Memcopies (D2H, H2D) asynchronous with CPU
Ability to concurrently execute a kernel and a memcopy
Stream = sequence of operations that execute in issue-order on
GPU
Operations from different streams may be interleaved
A kernel and memcopy from different streams can be overlapped
Overlap kernel and memory copy
Requirements:
D2H or H2D memcopy from pinned memory
Kernel and memcopy in different, non-0 streams
Code:
cudaStream_t   stream1, stream2;
cudaStreamCreate(&stream1);
cudaStreamCreate(&stream2);

cudaMemcpyAsync( dst, src, size, dir, stream1 );
kernel<<<grid, block, 0, stream2>>>(…);
potentially
overlapped
Call Sequencing for Optimal Overlap
CUDA calls are dispatched to the hw in the sequence they were issued
Fermi can concurrently execute:
Up to 16 kernels
Up to 2 memcopies, as long as they are in different directions (D2H and H2D)
A call is dispatched if both are true:
Resources are available
Preceding calls in the same stream have completed
Scheduling:
Kernels are executed in the order in which they were issued
Threadblocks for a given kernel are scheduled if all threadblocks for preceding
kernels have been scheduled and there still are SM resources available
Note that if a call blocks, it blocks all other calls of the same type behind it,
even in other streams
Type is one of { kernel, memcopy}
Stream Examples (current HW)
K1,M1,K2,M2: K1
K1 M1
M1 K2
K2 M2
M2
K1,K2,M1,M2: K1
K1 M1
M1 K2
K2 M2
M2
K1,M1,M2: K1
K1 M1
M1 M2
M2
K1,M2,M1: K1
K1 M1
M1 M2
M2
K1,M2,M2: K1
K1 M2
M2 M2
M2
Time
K:
Kernel
M:
Memcopy
Integer: Stream ID
More on Dual Copy
Fermi is capable of duplex communication with the host
PCIe bus is duplex
The two memcopies must be in different streams, different directions
Not all current host systems can saturate duplex PCIe bandwidth:
Likely issues with IOH chips
If this is important to you, test your host system
Duplex Copy: Experimental Results CPU
CPU--00  IOH
IOH  X58
X58  DRAM
DRAM  GPU
GPU--00  CPU
CPU--00  IOH
IOH  D36
D36  DRAM
DRAM  GPU
GPU--00  CPU
CPU--11  DRAM
DRAM
10.8 GB/s
7.5 GB/s
QPI, 6.4 GT/s
25.6 GB/s
3xDDR3, 1066 MHz
25.8 GB/s
PCIe, x16
16 GB/s
Duplex Copy: Experimental Results CPU
CPU--00  IOH
IOH  X58
X58  DRAM
DRAM  GPU
GPU--00  CPU
CPU--00  IOH
IOH  D36
D36  DRAM
DRAM  GPU
GPU--00  CPU
CPU--11  DRAM
DRAM
10.8 GB/s
11 GB/s
QPI, 6.4 GT/s
25.6 GB/s
3xDDR3, 1066 MHz
25.8 GB/s
PCIe, x16
16 GB/s
Unified Virtual Addressing
No UVA: Multiple Memory Spaces

UVA : Single Address Space
System
Memory CPU
CPU GPU0
GPU0
GPU0
Memory GPU1
GPU1
GPU1
Memory
PCI-e
0x0000
0xFFFF
0x0000
0xFFFF
0x0000
0xFFFF
System
Memory CPU
CPU GPU0
GPU0
GPU0
Memory GPU1
GPU1
GPU1
Memory
PCI-e
0x0000
0xFFFF
Easier to Program with Single Address Space
Summary
Kernel Launch Configuration:
Launch enough threads per SM to hide latency
Launch enough threadblocks to load the GPU
Global memory:
Maximize throughput (GPU has lots of bandwidth, use it effectively)
Use shared memory when applicable (over 1 TB/s bandwidth)
GPU-CPU interaction:
Minimize CPU/GPU idling, maximize PCIe throughput
Use analysis/profiling when optimizing:
“Analysis-driven Optimization” part of the tutorial following

Questions?


Abstract

Over the past decade, Graphics Processing Units (GPUs) have revolutionized high-performance computing, playing pivotal roles in advancing fields like IoT, autonomous vehicles, and exascale computing. Despite these advancements, efficiently programming GPUs remains a daunting challenge, often relying on trial-and-error optimization methods. This paper introduces an optimization technique for CUDA programs through a novel Data Layout strategy, aimed at restructuring memory data arrangement to significantly enhance data access locality. Focusing on the dynamic programming algorithm for chained matrix multiplication—a critical operation across various domains including artificial intelligence (AI), high-performance computing (HPC), and the Internet of Things (IoT)—this technique facilitates more localized access. We specifically illustrate the importance of efficient matrix multiplication in these areas, underscoring the technique’s broader applicability and its potential to address some of the most pressing computational challenges in GPU-accelerated applications. Our findings reveal a remarkable reduction in memory consumption and a substantial 50% decrease in execution time for CUDA programs utilizing this technique, thereby setting a new benchmark for optimization in GPU computing.

Keywords

Data Layout Optimization,  CUDA Performance Optimization,  GPU Memory Optimization,  Dynamic Programming,  Matrix Multiplication,  Memory Access Pattern Optimization in CUDA

Share and Cite:

1. Introduction

Graphics Processing Units (GPUs) have become ubiquitous accelerators in modern computing systems, offering tremendous parallel processing capabilities. Today, a single GPU provides thousands of compute cores capable of delivering teraflops of computational power, making GPU accelerator cards increasingly deployed in everything from mobile devices to cloud servers for a wide range of applications including artificial intelligence, scientific computing, and graphics [1] [2] . Despite these advancements, designing preferment and efficient GPU code is fraught with programming complexities and architectural constraints, particularly in threading, memory access, and parallelism management.

A critical challenge in leveraging GPU capabilities is managing data movement and organization on systems where there are processing-speed mismatches across components. For GPUs, which feature wide vector units and high memory bandwidth, disorganized or sparse data access patterns can incur high latency, leading to inefficiencies as memory controllers struggle to keep pace [3] . Optimizing data layout and access patterns is thus essential for feeding compute units efficiently and maximizing floating-point throughput. Another significant concern is Amdahl’s law, which posits that the speedup potential from parallel hardware is limited by the serial portions of code, indicating that various overheads such as kernel launch latency, host-to-device transfers, and synchronization delays can undermine the benefits of extensive parallelism. The efficient execution of matrix multiplication operations is a cornerstone in various computational domains, including deep learning, scientific simulations, and big data analytics. Optimizing these operations on GPUs can unlock significant performance gains, enabling faster training of neural networks, more accurate simulations, and accelerated data processing pipelines.

In this paper, we present a comprehensive approach aimed at addressing the aforementioned challenges, highlighting the efficacy of our proposed data layout optimization mechanism through the application of matrix multiplication. By focusing on the chained matrix multiplication (CMM) problem, recognized for its importance in various fields such as machine learning, physics simulations, and graphics, we develop and implement a dynamic programming algorithm optimized for CMM, utilizing CUDA code. The integration of a data layout transformation demonstrates a substantial improvement in memory locality and coalescing during parallel processing, underscoring the effectiveness of our approach [4] . Additionally, to further enhance performance, we explore additional parallelization techniques at the data level, including parallel diagonal computation and 2D thread block mapping, to maximize fine-grained concurrency. At the task level, we leverage streams and events to enable concurrent execution of multiple problem instances, optimizing the overlap between data transfers and computational processes, thereby further refining our optimization strategy.

This paper’s principles and techniques serve as a case study for unlocking the full potential of modern parallel accelerators. As the adoption of heterogeneous and GPU-based high-performance computing continues to grow rapidly, the programming practices and optimization strategies we discuss will be essential for harnessing the benefits of this technology [5] . Our work addresses key optimization challenges around parallelism management, data organization, and orchestration strategy [6] , offering insights that can be applied to adapt and implement various complex workloads on massively parallel processors.

2. Related Work

Prior work has developed optimizations for irregular data access [7] , data layout selection [8] , and communication reducing techniques when mapping algorithms onto GPU systems. While these existing approaches have demonstrated varying degrees of success, they often overlook the intricate interplay between memory access patterns, data layout, and the underlying GPU architecture. Additionally, many techniques are tailored to specific application domains or workloads, limiting their broader applicability. Our work aims to address these limitations by proposing a novel Data Layout strategy that restructures memory data arrangement to enhance locality and coalescing, thereby optimizing performance across a wide range of GPU-accelerated applications. Li et al. [9] proposed a simple yet effective data layout arbitration framework that automatically picks up the beneficial data layout for different DNNs under different pruning schemes. The proposed framework is built upon a formulated cache estimation model. Experimental results indicate that their approach is always able to select the most beneficial data layout and achieves the average training performance improvement with 14.3% and 3.1% compared to uniformly using two popular data layouts.

Zhenkun et al. [10] proposed a system dubbed Distributed Sampling and Pipelining (DSP) for multi-GPU GNN training. DSP adopts a tailored data layout to utilize the fast NVLink connections among the GPUs, which stores the graph topology and popular node features in GPU memory. For efficient graph sampling with multiple GPUs, they introduced a collective sampling primitive (CSP), which pushes the sampling tasks to data to reduce communication. They also design a producer-consumer-based pipeline, which allows tasks from different mini-batches to run congruently to improve GPU utilization. They compare DSP with state-of-the-art GNN training frameworks, and the results show that DSP consistently outperforms the baselines under different datasets, GNN models, and GPU counts. The speedup of DSP can be up to 26x and is over 2x in most cases. Wan et al. [11] introduced two online data layout reorganization approaches for achieving good tradeoffs between read and write performance. They demonstrated the benefits of using two approaches for the ECP particle-in-cell simulation WarpX, which serves as a motif for a large class of important Exascale applications. They showed that by understanding application I/O patterns and carefully designing data layouts they increased read performance by more than 80%.

Stoltzfus and Emani [12] proposed a machine learning-based approach to build a classifier to determine the best class of GPU memory that will minimize GPU kernel execution time. This approach utilizes a set of performance counters obtained from profiling runs along with hardware features to generate the trained model. They evaluated their approach on several generations of NVIDIA GPUs, including Kepler, Maxwell, Pascal, and Volta on a set of benchmarks. Their results showed that the trained model achieves prediction accuracy of over 90%. Majeti et al. Zhong et al. [13] introduce a new graph format with a data layout such that it supports coalesced access. Despite these promising results, existing optimization techniques for CUDA programs have inherent limitations. Memory bandwidth constraints, latency, and the non-uniform memory access (NUMA) architecture of GPUs may limit the applicability or performance benefits of these techniques in some scenarios. Furthermore, the dynamic nature of data access patterns in certain applications could reduce the effectiveness of static data layout optimizations.

3. Data Layout Technique

To optimize data access for parallel I/O, a data layout technique has been proposed and developed. This technique is successful in reducing the execution time of CUDA kernels and reducing their memory consumption. One of the types of data layout techniques implemented in this article is related to changing the arrangement of data in memory to improve memory access patterns and locality. The underlying principle of our Data Layout strategy is to restructure the memory layout of data structures, such as matrices, to align with the access patterns of the target algorithm. By storing elements that are accessed consecutively in contiguous memory locations, we can enhance spatial locality and leverage hardware caching mechanisms more effectively. This approach reduces cache misses and improves coalesced memory accesses, leading to more efficient utilization of the GPU’s memory subsystem.

Consider the case of matrix multiplication, a critical operation in various domains. Traditionally, matrices are stored in row-major or column-major order, which may not be optimal for certain access patterns. Our Data Layout technique explores alternative storage formats, such as diagonal-based or blocked layouts, to improve memory access locality for the specific algorithm being executed. Here we want to implement this technique on a matrix of numbers. Before we use the data layout on the matrix, it is important to understand the various data layout patterns. In a matrix, there are different ways in which data is written to memory, including row-based and column-based data layouts:

3.1. Row-Based Storage

In row-based storage, data for a single row of a table is stored consecutively on memory. This means that all the columns of a given row are stored together, which can make it efficient for operations that need to access an entire row of data at once.

3.2. Column-Based Storage

In column-based storage, each column of a matrix is stored consecutively on memory. This can be more efficient for operations that only need to access a subset of the columns in a matrix. Despite the promising results, certain inherent limitations of data layout optimization techniques in GPU programming warrant consideration. Issues such as memory bandwidth constraints, latency, and the non-uniform memory access (NUMA) architecture of GPUs may limit the applicability or performance benefits of these techniques in some scenarios. Furthermore, the dynamic nature of data access patterns in certain applications could reduce the effectiveness of static data layout optimizations.

4. Case Study: Chained Matrix Multiplication Problem

We address the problem of chained matrix multiplication (CMM), a cornerstone in computing, with a dynamic programming algorithm. This algorithm optimizes the order of matrix multiplication operations, a task crucial for minimizing computational workloads. The goal is to develop an algorithm that determines the optimal order for multiplying n matrices, as the optimal order depends only on the matrix dimensions. Consider the multiplication of the following n matrices:

A

      1

     ×

      A

      2

     ×

      A

      3

     ×

     ⋯

     ×

      A

      n

The number of multiplications required to multiply two matrices An×m × Bm×k is n × m × k times. Matrix multiplication is an associative operation, meaning that the order in which we multiply do not matter. Therefore, the number of multiplications is dependent on the order of matrix multiplication. For example, for four matrix multiplication of A20×2 × B2×30 × C30×12 × D12×8 (n = 4), there are five different orders in which we can multiply four matrices, each possibly resulting in a different number of elementary multiplications:

• A(B(CD)): 30 × 12 × 8 + 2 × 30 × 8 + 20 × 2 × 8 = 3680

• (AB)(CD): 20 × 2 × 30 + 30 × 12 × 8 + 20 × 30 × 8 = 8880

• A((BC)D): 2 × 30 × 12 + 2 × 12 × 8 + 20 × 2 × 8 = 1232

• ((AB)C)D: 20 × 2 × 30 + 20 × 30 × 12 + 20 × 12 × 8 = 10320

• (A(BC))D: 2 × 30 × 12 + 20 × 2 × 12 + 20 × 12 × 8 = 3120

The third order is the optimal order for multiplying the four matrices. In this problem, the goal is to develop an algorithm that determines the optimal order for multiplying n matrices. The optimal order depends only on the dimensions of the matrices. Therefore, besides n, these dimensions would be the only input to the algorithm.

5. Serial Algorithm by Dynamic Programming Method

In this section, the dynamic programming solution for the problem of chain multiplication of matrices is described. We first present the serial dynamic programming solution for the CMM problem [14] , which avoids redundant calculations by breaking down the problem into subproblems and storing the results in a matrix M. The provided code is a serial implementation, without taking advantage of parallel processing.

Input: int n, int dim [

     0

     ⋯

     n

 ]. Here, n is the number of matrices and dim contains the dimensions of the matrices. For instance, for A20×2 × B2×30 × C30×12 × D12×8, inputs are n = 4 and dim = [20, 2, 30, 12].

Output: int A[n + 1][n + 1], int M[n + 1][n + 1]. Here, M [i, j] is the minimum number of multiplications in the ith to jth matrix multiplication. Also, if A[i, j] = k(i ≤ k < j), then the optimal order of multiplication in the ith to jth matrix multiplication is (Ai × ∙∙∙ × Ak) × (Ak+1 × ∙∙∙ × Aj) and the optimal number of multiplications is M[1, n].

Algorithm: Inside each parenthesis, the multiplications are obtained according to the optimal order for the matrices inside the parentheses. Of these factorizations, the one that yields the minimum number of multiplications must be the optimal one. The number of multiplications for the kth factorization is the minimum number needed to obtain each factor plus the number needed to multiply the two factors. This means that it equals:

M

      [

      1

      ]

      [

      n

      ]

     =

       min

       1

       ≤

       k

       <

       n

      (

       M

        [

        1

        ]

        [

        k

        ]

       +

       M

        [

         k

         +

         1

        ]

        [

        n

        ]

       +

        d

        1

        d

        k

        d

        n

      )

To calculate the intermediate values, the formula is as follows:

M

      [

      1

      ]

      [

      n

      ]

     =

       min

       1

       ≤

       k

       <

       n

      (

       M

        [

        1

        ]

        [

        k

        ]

       +

       M

        [

         k

         +

         1

        ]

        [

        n

        ]

       +

        d

        1

        d

        k

        d

        n

      )

With:

M

      [

      i

      ]

      [

      i

      ]

     =

     0

     ,

     fori

     =

     0

     ⋯

     n

Calculations are performed diagonally, starting from the main diameter as shown in Figure 1.

Figure 1. The order of calculations in algorithm.

5.1. Serial Algorithm by Dynamic Programming Method for CMM Problem

This is an implementation of a dynamic programming solution for the Chain Matrix Multiplication (CMM) problem [14] . This problem involves finding the most efficient way to multiply a sequence of matrices together. The dynamic programming approach efficiently avoids redundant calculations by breaking down the problem into subproblems and storing the results of those subproblems in the matrix M. The provided code is a serial implementation, meaning it doesn’t take advantage of parallel processing. The CMM problem and its dynamic programming solution are commonly used in algorithmic optimization for matrix chain multiplication scenarios. The serial code of this algorithm is presented in Listing 1.

Listing 1. Serial version by dynamic programming method for CMM problem.

5.2. Parallel Version in CUDA C++

Considering that in dynamic programming algorithms, arrays are used to store data, there is a very good parallelization capability in them. Each data item in a diagonal of this matrix is calculated by a thread. In the first step, n threads set to zero the values of the main diameter. Then, at each step, one thread is set aside, so that finally, in the last round, only one thread calculates the value of M [1][n]. Considering that the values calculated in each diameter are dependent on the values of the previous diameters, therefore, at the end of each round, synchronization must be done between all the threads. This action is done through the __syncthreads() instruction. This instruction only synchronizes the threads in a block. So, we are only able to use one block in calculations. Global synchronization between all blocks of a kernel has not been implemented in the CUDA programming model, and no instructions have been published for it by NVIDIA. Therefore, the kernel configuration in this program is <<<1, n>>>.

In the CUDA version, the matrix is converted to a one-dimensional array. In the CUDA programming guide [15] , it is recommended to use a one-dimensional array instead of a matrix in the kernel. So, M[i][j] in the matrix is mapped to M[(n + 1) × i + j] in the one-dimensional array.

Kernal launching in this program is:

CMM_CUDA_kernel<<<1, n>>>(dev_dim, dev_m, dev_result, n).

dev_dim, which is sent as an argument to the kernel, is a one-dimensional array of the dimensions of the matrix. dev_m is also the matrix M that is used for calculations and has the same function as the serial version. Before kernel launching, the data must be transferred from the main memory of the CPU to the global memory of the GPU. After the execution of the kernel, the results should be transferred in the reverse direction.

6. Tuning of Algorithm by Data Layout Technique

In the serial algorithm, the computation in the matrix M is done diagonally. However, in the C++ language, matrices are stored as rows in memory. Therefore, thread accesses to memory are not consecutive, reducing locality and increasing cache misses and uncoalesced accesses to global memory, which decreases performance.

To address this issue, we apply the Data Layout technique by storing the elements of each diagonal together in memory, as described in Section 3. This change requires modifications to the indices accessed in the algorithm. By restructuring the data layout, we increase locality and improve the probability of cache hits and coalesced accesses, leading to significant performance improvements. The proposed Data Layout strategy is based on the principle of organizing data in a way that aligns with the access patterns of the program. By storing related data elements consecutively in memory, we can increase the likelihood of cache hits and coalesced memory accesses, reducing memory bandwidth bottlenecks and improving overall throughput. The use of this technique requires changes in the codes and the indices accessed in the algorithm must be changed. We explain this technique with an example. For instance, consider the data of a matrix as shown in Figure 2.

Figure 2. Initial Matrix data.

We only describe an upper triangular matrix because this type of matrix is also used in the chained multiplication problem. Figure 3 illustrates the typical way to store this matrix in the memory.

Figure 3. Data layout in memory.

Figure 4 exhibits how our proposed data layout technique that we used in this problem stores the data of the above matrix in the memory.

Figure 4. Proposed data layout technique in memory.

Therefore, Locality increases strongly and increases the possibility of cache hit and coalesced accesses. We applied this technique to the algorithm of chained multiplication of matrices and obtained promising results which are reported in the following section.

7. Additional Parallelization Techniques

7.1. Data Level Parallelism

To further exploit parallelism in the chain matrix multiplication algorithm, we apply techniques to partition the data at a finer granularity.

Parallelizing Diagonal Computations. Our existing implementation maps one GPU thread to compute each element along the diagonal. We modify this to use multiple threads to compute each element by decomposing the computation in dimensions as shown in Listing 2.

Listing 2. Parallelizing diagonal computation.

We use a (16 × 16) thread block, enabling (256) threads to cooperate in computing each matrix element. This adds finer-grained parallelism within each

diagonal. On our GPU with (128) CUDA cores per SM, this enables each SM to process 2 diagonal elements in parallel. The (16 × 16) block also improves memory access patterns. Compared to the one thread per element approach, this parallel diagonal computation reduces kernel time by 41%

2D Thread Block Decomposition. We also decompose the total computation into 2D thread blocks, assigning each diagonal across multiple blocks as specified in Listing 3.

Listing 3. Block decomposition.

This spreads the work of a diagonal over more GPU cores for greater parallelism. This allows more SMs and CUDA cores to operate on a diagonal in parallel. With N/16 blocks, more SMs participate, and overall parallelism improves. Using 2D thread blocks gives a 23% kernel speedup over the parallel diagonal method alone.

7.2. Task Level Parallelism

To overlap computation and transfers between the CPU and GPU, we leverage streams, events, and concurrency [16] as illustrated in Listing 4.

Listing 4. Task level parallelism.

We also parallelize across multiple independent problem instances by allocating separate streams and CUDA contexts for each instance. This enables entirely concurrent execution. The streams and asynchronous calls prevent these operations from blocking each other. This improves GPU utilization and end-to-end runtime. With 2 streams per instance, we get up to 4× speedup with 4 problems run in parallel.

8. Evaluation

We conducted a series of experiments to evaluate the performance of our proposed Data Layout strategy compared to existing optimization techniques, focusing on the chained matrix multiplication (CMM) problem. The experiments were performed on a system equipped with an NVIDIA Tesla V100 GPU, utilizing the CUDA programming model. We implemented our optimization technique and compared it against conventional memory layouts such as row-major and column-major order, as well as other state-of-the-art optimization strategies proposed in prior literature.

We leverage cuda Event tool for profiling the execution time of programs, which provides very good accuracy compared to the clock() function. To use this tool for recording the execution time, we use the solution shown in Listing 5.

Listing 5. Recording execution time.

Table 1 presents the execution times for the serial CPU dynamic programming algorithm, the baseline CUDA GPU parallel implementation, and the CUDA version optimized with the Data Layout technique, across different numbers of input matrices (n = 1016 to 1024). Note that it is not possible to run the program for n > 1024, as the maximum number of threads per block is 1024. All times are measured in milliseconds. As shown in Figure 5, the CUDA implementation demonstrates more than 2× speedup compared to the serial algorithm across all matrix sizes. For n = 1024 matrices, the serial algorithm takes 1287 ms, while the CUDA implementation requires only 542 ms, achieving a 2.4× runtime improvement by harnessing the parallel processing power of the GPU.

Table 1. Execution time comparison.

Figure 6 provides deeper insight into the performance trends for smaller input sizes in the matrix chain multiplication problem. We plot execution times for a range of n = 10 to 400 matrices to examine why the serial CPU implementation outperforms the CUDA GPU code at very small n. The breakeven point where CUDA becomes faster occurs around 218 matrices. Below this threshold, the parallel CUDA overheads of copying memory between host and device as well as launching computational kernels overwhelm the relatively minor parallelism benefits for small inputs.

Figure 5. Execution time comparison between serial and CUDA version.

However, beyond n = 218 matrices, the runtime of the serial algorithm grows super linearly due to its algorithmic complexity of O(n3). In contrast, CUDA runtime grows roughly linearly thanks to exploiting parallel hardware. Ultimately, this allows the CUDA performance curve to cross below the serial line. Profiling shows kernel launch overheads are relatively fixed at around 0.4 ms, while serial algorithm runtime scales worse than linearly. This highlights why CUDA provides increasing returns as the problem size grows—parallel hardware continues delivering a fixed amount of extra throughput, surpassing serial execution.

Figure 6. Execution time comparison between serial and CUDA version.

Figure 7 compares the execution time between our baseline naive CUDA implementation and the version optimized with the data layout transformation technique. The optimized CUDA code accesses matrix data with significantly improved locality and coalescing, providing up to a 2× faster runtime, with an average speedup exceeding 1.8×. Performance gains are consistent across all input sizes, demonstrating the effectiveness of our Data Layout technique in accelerating memory access patterns.

Compared to previous optimization approaches that relied on compiler auto-vectorization or manual code transformations, our Data Layout strategy offers a more systematic and architecture-aware solution. By explicitly restructuring the memory layout, we can ensure optimal data access patterns tailored to the GPU’s memory hierarchy, leading to superior performance gains.

This runtime comparison shows the clear performance benefits of optimizing memory access patterns on the GPU using our data layout transformation. Rather than relying on the default row-major matrix storage, we rearrange elements to store diagonals consecutively in memory. This matches the access pattern of our dynamic programming algorithm, which iterates diagonally through the matrix. Laying out data to improve locality directly accelerates these memory reads and writes. We see execution time reduced by over 50%, with the optimized CUDA implementation running over 1.8× faster across all input sizes. At 1024 matrices, runtime drops from 542ms in the baseline CUDA code down to just 272 ms with data layout improvements. By enhancing memory coalescing and exploiting caching, the GPU no longer wastes cycles waiting on scattered uncoalesced memory accesses. This demonstrates data layout changes can unlock substantial performance gains by alleviating bottlenecks related to noncontiguous data access. The optimization builds on earlier CUDA speedups for a combined 4× total improvement over the original serial algorithm.

Figure 7. Comparison of CUDA version and Optimized CUDA version.

Figure 8 depicts the speedup attained by our optimized CUDA GPU algorithm with the data layout strategy, relative to the performance of the baseline naive CUDA implementation. We observe an average 1.9× speedup, reaching as high as 2.04× for some matrix input sizes. This demonstrates the effectiveness of improved memory locality in unlocking performance, cutting execution time by up to 50%.

Figure 8. Comparison of CUDA version and Optimized CUDA version.

9. Discussion and Future Work

The remarkable performance improvements achieved by our Data Layout strategy highlight its potential for unlocking the true computational power of GPUs across a wide range of applications. While our case study focused on chained matrix multiplication, the underlying principles of our approach are applicable to various algorithms and data structures that exhibit non-contiguous or irregular memory access patterns. Despite the promising results, certain inherent limitations of our Data Layout technique warrant consideration. Issues such as memory bandwidth constraints, latency, and the non-uniform memory access (NUMA) architecture of GPUs may limit the applicability or performance benefits in some scenarios. Furthermore, the dynamic nature of data access patterns in certain applications could reduce the effectiveness of static data layout optimizations. Looking ahead, our future work will focus on exploring the scalability and effectiveness of our Data Layout strategy in large-scale GPU clusters and cloud environments. By conducting extensive experiments across distributed systems, we aim to provide deeper insights into the potential challenges and opportunities of our approach in these advanced computing paradigms. Additionally, we plan to investigate the integration of our technique with other optimization strategies, such as dynamic data layout transformations and adaptive memory management, to further enhance performance and mitigate the limitations mentioned above.

10. Conclusions

In this study, we presented a novel Data Layout strategy for optimizing CUDA programs, particularly focusing on dynamic programming algorithms for chained matrix multiplication. By restructuring memory data arrangement to improve locality and coalescing, our approach achieved significant performance enhancements, reducing execution time by up to 50% and memory consumption compared to the baseline implementation. The effectiveness of our technique underscores the importance of architecture-aware optimizations in unlocking the full potential of GPU-accelerated applications. As the adoption of heterogeneous and GPU-based computing continues to grow rapidly across various domains, the principles and strategies discussed in this work will be instrumental in harnessing the benefits of these powerful parallel architectures.

While our findings are promising, we acknowledge the limitations and challenges associated with our approach and emphasize the need for further research to extend its applicability and address potential scalability concerns. By continuing to explore innovative optimization techniques and leveraging the synergies between hardware and software, we can pave the way for more efficient and high-performance GPU computing solutions.

It’s important to acknowledge that the scope of our experiments was limited by time constraints, restricting our ability to conduct a more extensive investigation. Future studies will aim to address this limitation by allocating more time for rigorous experimentation and analysis.

Acknowledgements

This work is supported by NSF award #2348330.

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

[1]

                                        Harris, M. (2007) Optimizing Parallel Reduction in CUDA. Nvidia Developer Technology, 2, 70.

                                        [2]

                                        Hong, S. and Hyesoon, K. (2010) An Integrated GPU Power and Performance Model. ACM SIGARCH Computer Architecture News, 38, 280-289. https://doi.org/10.1145/1816038.1815998

                                        [3]

                                        Volkov, V. and Demmel, J.W. (2008) Benchmarking GPUs to Tune Dense Linear Algebra. Proceedings of the 2008 ACM/IEEE Conference on Supercomputing, Austin, 15-21 November 2008. https://doi.org/10.1109/SC.2008.5214359

                                        [4]

                                        Wu, Y.N., Tsai, P.-A., Muralidharan, S., Parashar, A., Sze, V. and Emer, J. (2023) HighLight: Efficient and Flexible DNN Acceleration with Hierarchical Structured Sparsity. Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture, Association for Computing Machinery, New York, NY, USA, 1106-1120. https://doi.org/10.1145/3613424.3623786

                                        [5]

                                        Liu, G., et al. (2018) A Scalable Parallel Method for Large-Scale Matrix Computations. The Journal of Supercomputing, 74, 6641-6656.

                                        [6]

                                        Peled, L., Mannor, S., Weiser, U. and Etsion, Y. (2015) Semantic Locality and Context-Based Prefetching Using Reinforcement Learning. ACM SIGARCH Computer Architecture News, 43, 285-297. https://doi.org/10.1145/2872887.2749473

                                        [7]

                                        Aldinucci, M., Drocco, M., Mastrostefano, F. and Vanneschi, M. (2018) Hardware-Conscious Autonomic Management of Distributed Workflows. International Conference on Algorithms and Architectures for Parallel Processing, Springer, Cham, 27-31 August 2018, 343-359.

                                        [8]

                                        Ballard, G., Zheng, G., Demmel, J. and Yelick, K. (2017) An Efficient and Generic Event-Based Profiling Framework for GPU Architectures. IEEE Transactions on Parallel and Distributed Systems, 29, 169-182.

                                        [9]

                                        Li, B.Y., et al. (2022) Optimizing Data Layout for Training Deep Neural Networks. Companion Proceedings of the Web Conference 2022, New York, April 2022, 548-554. https://doi.org/10.1145/3487553.3524856

                                        [10]

                                        Cai, Z., Hu, L., Shi, B., Chen, Y., Hu, C. and Tang, J. (2023) DSP: Efficient GNN Training with Multiple GPUs. Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, Montreal, February 2023, 392-404. https://doi.org/10.1145/3572848.3577528

                                        [11]

                                        Wan, L.P., et al. (2022) Improving I/O Performance for Exascale Applications through Online Data Layout Reorganization. IEEE Transactions on Parallel and Distributed Systems, 33, 878-890. https://doi.org/10.1109/TPDS.2021.3100784

                                        [12]

                                        Stoltzfus, L., et al. () Data Placement Optimization in GPU Memory Hierarchy Using Predictive Modeling. Proceedings of the Workshop on Memory Centric High Performance Computing, Dallas, November 2018, 45-49. https://doi.org/10.1145/3286475.3286482

                                        [13]

                                        Zhong, J.L. and He, B.S. (2014) Medusa: Simplified Graph Processing on GPUs. IEEE Transactions on Parallel and Distributed Systems, 25, 1543-1552. https://doi.org/10.1109/TPDS.2013.111

                                        [14]

                                        Neapolitan, R. and Naimipour, K. (2008) Foundations of Algorithms Using C   Pseudocode. 3rd Edition, Jones and Bartlett Publishers, Inc., Sudbury, USA.

                                        [15]

                                        NVIDIA (2023) CUDA C   Programming Guide (Version 12.2). https://docs.nvidia.com/cuda/archive/12.2.0/pdf/CUDA_C_Best_Practices_Guide.pdf

                                        [16]

                                        Segura, A., Arnau, J.-M. and González, A. (2019) SCU: A GPU Stream Compaction Unit for Graph Processing. Proceedings of the 46th International Symposium on Computer Architecture, Phoenix, Arizona, 22-26 June 2019, 424-435. https://doi.org/10.1145/3307650.3322254

Journals Menu

Copyright © 2025 by authors and Scientific Research Publishing Inc.

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.

Home

About SCIRP

Service

Policies


On this page

GPU MODE Lecture 6: Optimizing Optimizers in PyTorch

Christian Mills

September 2, 2024

Introduction

Optimization Analogy: Towing Cars

Optimizer Basics and Optimization Levels

parameter = parameter - learning_rate * gradient

Implementation: Processes parameters one by one in a for loop.

Simplified Example:

for param in params:
    # Retrieve necessary data for the current parameter
    # Perform operations (add, multiply, lerp, etc.)
    # Update the parameter

Visualization: Each parameter update is a sequence of operations (gray circles), represented as a column. M operations per parameter, N parameters total, resulting in M x N operations.

Implementation: PyTorch’s current default; operates on entire parameter lists at once using vectorized operations.

Simplified Example:

# Add a constant to all parameters in the list
# Multiply all parameters by a constant
# ... other operations

Visualization: Each operation (blue circles) is performed on all parameters simultaneously. M operations total.

Multi-Tensor Apply: The Powerhouse

Standard Add (Simplified):

__device__ void add_kernel(float* self, float* other, float* res, float alpha=1);

ForEach Add (Challenge): How would you design a CUDA kernel signature to handle tensor lists?

Attempt 1: Passing Standard Vector (Failed)

Attempt 2: Passing Pointers to Pointers (Failed)

Attempt 3: Passing by Chonky Boy (Partially Successful)

Idea: Pass tensor data pointers by value using a struct.

Implementation:

Outcome: Works initially, but encounters issues with the kernel argument space limit.

Kernel Argument Space Limit: The kernel argument space has a maximum size of 4 kilobytes.

Problem: If the struct containing tensor pointers exceeds 4 kilobytes, only a portion of the struct gets passed to the kernel, leading to illegal memory access when accessing pointers beyond the limit.

Repro Example:

params = [torch.rand(2,3, device="cuda") for _ in range(N)]
torch._foreach_norm(params, ord=1)
torch.cuda.synchronize()

Observation: Illegal memory access occurs when the number of tensors exceeds 423.

Conclusion: The struct approach works as long as the number of tensor pointers does not exceed the 4 kilobyte limit.

Solution 1: Batching (Current Implementation)

Solution 2: Revisiting Pointers to Pointers with Memcpy

Solution 3: Unified Memory (Future Exploration)

Fused Optimizers and Multi-Tensor Apply

Torch Compile and the Future of Fused Optimizers

Torch Compile (Inductor): PyTorch’s compiler that excels at vertical fusion of operations.

Potential: Automate the vertical fusion of optimizer operations, eliminating the need for handwritten CUDA kernels.

Benefits:

Current Status:

Example:

optimizer = torch.optim.AdamW(model.parameters())

@torch.compile(fullgraph=False)
def compiled_step():
    optimizer.step()

# ... training loop
compiled_step()

Limitations:

Future Directions:

Q&A Session

Obtaining Triton Kernel Code

Visualizing Kernel Graphs

Compile Time Dependency on Number of Tensors

Caching Compiled Results

Memory Management with Structs Containing Pointers

Memcpy Size Limit

Memcpy Direction Argument

Device-to-Device Copy

I’m Christian Mills, a deep learning consultant specializing in practical AI implementations. I help clients leverage cutting-edge AI technologies to solve real-world problems.

Interested in working together? Fill out my Quick AI Project Assessment form or learn more about me.

Content licensed under CC BY-NC-SA 4.0

© 2025 Christian J. Mills

Code samples licensed under the MIT License