Beating cuBLAS in Single-Precision General Matrix Multiplication

Jan 12, 2025
      • Aman Salykov

This project is inspired by the outstanding works of Andrej Karpathy, George Hotz, Scott Gray, Horace He, Philippe Tillet, Jeremy Howard, Lei Mao and the best CUDA hackers from the GPU MODE community (Discord server). A special thanks to Mark Saroufim and Andreas Köpf for running GPU MODE and all you’ve done for the community.

The code is available at sgemm.cu. This article complements my blog post, which covers the implementation of FP32 matrix multiplication that outperforms BLAS libraries on modern Intel and AMD CPUs. Today we’ll walk through a GPU implementation of SGEMM (Single-precision GEneral Matrix Multiply) operation defined as C := alpha*A*B + beta*C. The blog delves into benchmarking code on CUDA devices and explains the algorithm’s design along with optimization techniques. These include inlined PTX, asynchronous memory copies, double-buffering, avoiding shared memory bank conflicts, and efficient coalesced storage through shared memory. I’d also like to mention that the high-level algorithm design used in this project was developed by the excellent engineers at NVIDIA and has been extensively studied in prior works on cuBLAS and CUTLASS. My main contribution was translating it into efficient CUDA/PTX code. The goal of this project wasn’t to build an SGEMM that would magically outperform cuBLAS on all GPUs and all matrix sizes. This is especially pointless, given the open-sourced, lightweight CUTLASS library. Instead, the project primarily targets CUDA learners and aims to bridge the gap between the SGEMM implementations explained in books/blogs and those used in NVIDIA’s BLAS libraries. While the implementation is expected to deliver high performance on Ada/Ampere/Volta/Turing devices, it was specifically fine-tuned for and tested on a local NVIDIA RTX 3090 (=GA102 chip: RTX 3080, A10, A40, A6000). The achieved performance is shown below, comparing results with locked and unlocked GPU core frequencies against cuBLAS and Simon Boehm’s highly cited work (used in llamafile, aka tinyBLAS). I plan to continue publishing educational content on high-performance kernels used in AI/ML. Let me know what topics you’d like to see next! Projects currently in development: beating NVIDIA on Tensor Cores, Stream-K GEMM, FlashAttention, xLSTM. If you enjoy educational content like this and would like to see more, please share this article. Your feedback would be greatly appreciated!

P.S. Please feel free to get in touch if you are interested in collaborating. My contact information is available on the homepage.

1. Introduction

I clearly remember Andrej’s post on the current state of the existing cuda learning materials vs. cuda code used in high-performance libraries:

Indeed, when it comes to SGEMM implementations, there are some excellent educational blog posts, such as

that break down, step by step, how to optimize a CUDA matmul kernel. However, in terms of achieved performance, none of them come close to matching the speed of cuBLAS or CUTLASS, especially when using recent CUDA versions and if benchmarked properly. From my experiments, these implementations achieve 50-70% of cuBLAS’ performance at best. Additionally, I found the explanations in both blog posts a bit overcomplicated in the final optimization steps. Nevertheless, I still think these resources are great for anyone starting with CUDA programming since they provide good foundational knowledge.

On the other hand, I’ve seen some really fast SGEMM implementations with cuBLAS-level performance:

The problem is that they are undocumented, difficult to find and understand, especially for a CUDA beginner. A similar problem exists with CUTLASS. While it is highly performant, there is a lack of introductory or educational materials explaining how it is internally designed and implemented in efficient CUDA/PTX. Another notable project is MaxAs, an assembler for the Maxwell architecture developed over a decade ago by Scott Gray. This tool enables programming directly in SASS (the assembly language for NVIDIA GPUs), allowing direct communication with the hardware instead of relying on the hardware-agnostic CUDA/PTX. Using MaxAs, Scott wrote an SGEMM implementation that achieved around 98% of the GM204 chip’s theoretical maximum FLOPS, surpassing cuBLAS by an average of 5%. While the results are impressive, programming in SASS is inflexible and requires deep understanding of the underlying hardware. Furthermore, with significant advancements in the compiler since then, programming directly in SASS is only advantageous in exceptional cases (for example, if you build tinygrad). CUTLASS achieves performance on par with cuBLAS across various GPU architectures and matrix sizes using only CUDA/PTX code.

But can we actually exceed the cuBLAS barrier? In the following chapters, we will briefly review the high-level SGEMM design used in CUTLASS, and discuss how to translate this design into efficient CUDA/PTX. This guide assumes only a basic knowledge of the CUDA programming model and linear algebra. If you are new to CUDA programming, I strongly recommend starting with these short introductory articles:

Before we proceed with implementation, let’s talk about benchmarking code on NVIDIA GPUs - a topic often overlooked. Properly benchmarking code is just as important as the code itself, particularly when comparing different implementations.

2. How to Benchmark Code on CUDA Devices?

The most reliable way to measure kernel duration is by profiling with NVIDIA Nsight Compute and manually extracting performance data. To obtain deterministic and reproducible results, Nsight Compute automatically applies the following settings:

Alternatively, you can apply these settings manually and measure kernel duration at runtime without relying on external profilers. On Ubuntu, you can retrieve the base core clock frequency using:

nvidia-smi base-clocks

For instance, on an RTX 3090, the base core clock frequency is 1395 MHz. Next, you’ll need the memory clock frequencies, which work in combination with the base core clock:

nvidia-smi -q -d supported_clocks

From the list of supported frequencies, choose the fastest memory clock compatible with the base core frequency. Memory clock speeds are generally more stable than core clock speeds. To lock the clock frequencies and enable persistence mode, run the following commands:

sudo nvidia-smi --persistence-mode=1
# NVIDIA RTX 3090
sudo nvidia-smi --lock-gpu-clocks=1395
sudo nvidia-smi --lock-memory-clocks=9501

To reset the core and memory clock frequencies, you can use:

sudo nvidia-smi --reset-gpu-clocks
sudo nvidia-smi --reset-memory-clocks
sudo nvidia-smi --persistence-mode=0

GPU clock frequencies may drop due to the GPU’s thermal state, but for high-performance applications, throttling is often caused by power limits. Faulty hardware can also lead to throttling. It’s a good idea to monitor the GPU’s state at least during a test run. Use the following command to keep track of power draw, clock speeds, and throttling reasons in real time:

watch -n 0.1 nvidia-smi --query-gpu=power.draw,clocks.sm,clocks.mem,clocks_throttle_reasons.active --format=csv

A sample output might look like this:

308.50 W, 1395 MHz, 9501 MHz, 0x0000000000000000

The bit mask 0x0000000000000000 indicates no throttling, and the clocks are running at their maximum speeds. A value of 0x0000000000000001 indicates an idle state. Any other values suggest throttling is occurring. For a full list of bit mask values and their meanings, refer to the NvmlClocksThrottleReasons documentation.

Once you’ve locked the clock frequencies, you can measure the kernel duration directly in CUDA using CUDA events. Here’s an example:

cudaEvent_t start, stop;
cudaEventCreate(&start); cudaEventCreate(&stop);
float elapsed_time_ms = 0.0;

cudaEventRecord(start);
kernel<<<...>>>(...);
cudaEventRecord(stop);

cudaEventSynchronize(stop);
cudaEventElapsedTime(&elapsed_time_ms, start, stop);

For reliable measurements, multiple replay passes are typically used. In such cases, the GPU cache should be flushed before each kernel replay. This can be done using cudaMemsetAsync as shown in nvbench:

// Flush L2 cache
int dev_id{};
int m_l2_size{};
void* buffer;
checkCudaErrors(cudaGetDevice(&dev_id));
checkCudaErrors(cudaDeviceGetAttribute(&m_l2_size, cudaDevAttrL2CacheSize, dev_id));
if (m_l2_size > 0) {
    checkCudaErrors(cudaMalloc(&buffer, static_cast<std::size_t>(m_l2_size)));
    int* m_l2_buffer = reinterpret_cast<int*>(buffer);
    checkCudaErrors(cudaMemsetAsync(m_l2_buffer, 0, static_cast<std::size_t>(m_l2_size)));
    checkCudaErrors(cudaFree(m_l2_buffer));
}

Locking the clock frequencies to their base values is a reliable way to measure the speed of your kernel. However, in real-world scenarios, algorithms don’t typically run with locked clocks. To achieve optimal performance, your algorithm needs to be both fast and power-efficient. The less power your algorithm consumes, the higher the clock speeds your hardware can maintain. NVIDIA GPUs often reduce clock frequencies aggressively, well before hitting their power limits, which can significantly degrade application performance. To account for this, we benchmark our implementation under both locked and unlocked clock conditions, testing for both speed and power efficiency. In our benchmarks, we evaluate matrix sizes ranging from 1024 to 12,800 with a step size of 128. For each matrix size, we launch 1000*exp((-matsize+1024)/3100.0)) kernel replays and calculate the runtime as the average of the second half of the replays. For example, given matrix size problem m=n=k=4096, we run the sgemm 1000*exp((-4096 + 1024)/3100.0))=371 times and measure the average duration of the last 185 runs, ensuring the clocks have stabilized. This profiling strategy leads to consistent and reproducible results, even when GPU clocks are unlocked.

Avoid using WSL for performance measurements. To ensure accurate and reliable results, please use a native Linux environment.

3. Memory Layout

Without loss of generality in this implementation, we assume matrices are stored in row-major order. A matrix A with dimensions M x N is stored as contiguous array of length M*N. Elements A[row][col] are accessed via a 1D raw C pointer ptr[row*N + col] with 0<=col<=N-1 and 0<=row<=M-1. Matrix multiplication is denoted as $C=AB$, where the shapes of matrices $A, B, C$ are $M \times K, K \times N$ and $M \times N$, respectively.

To adapt this implementation for matrices stored in column-major order, simply swap the operands $A$ and $B$, because:

Here, $A, B, C$ are matrices stored in row-major order, while $A^\text{T}, B^\text{T}, C^\text{T}$ are the corresponding transposed matrices (i.e., stored in column-major order).

cuBLAS provides an API to calculate SGEMM:

cublasSgemm(m, n, k, A, lda, B, ldb, C, ldc); // simplified form

with m, n, k denote the matrix sizes $M, N, K$. The parameters lda, ldb, ldc are the leading dimensions of matrices $A, B, C$, respectively. The leading dimension is the length of the fastest-varying dimension when iterating over the matrix elements (i.e., the length of the first dimension). For matrices stored in row-major order, the leading dimension is usually the number of columns, so typically lda=k, ldb=n, ldc=n. However, this isn’t always the case. In scenarios where you need to compute a submatrix of a larger matrix, the leading dimension might be larger than the number of columns.

Matrices may be also padded with zeros to support vectorized memory loads or tensor cores. The vectorized load instructions allow to load multiple elements at once using just 1 instruction. Though the vectorized loads reduce total number of instructions and improve bandwidth utilization, they also impose alignment constraint on input data, so that the leading dimension must be divisible by 2 (for 64-bit loads) or 4 (for 128-bit loads). The figure below illustrate the case for 128-bit (=4 floats) loads.

Note how it’s impossible to load the elements of the first row without touching the elements of the next row if the leading dimension is not divisible by 4. Padding with zeros helps, but requires additional memory. Another solution would be to check at runtime if the leading dimension is divisible by 4. If it is - then use vectorized loads, if not - scalar loads. Additionally, zero padding was commonly used in the past to enable tensor core computations. For instance, in cuBLAS versions < 11, Tensor Core FP16 operations required m, n, k to be multiples of 8.

4. Parallel Thread Execution

The CUDA compilation trajectory of a .cu file looks as follows:

During Stage 1 CUDA code is compiled to PTX (parallel thread execution) instructions - intermediate high-level code, which can be considered as assembly for a virtual GPU architecture. Such a virtual GPU is defined entirely by the set of capabilities, or features, that it provides to the application. PTX doesn’t run on any real architecture, directly. It must be optimized and translated to native target-architecture instructions (Stage 2). NVIDIA provides a mechanism to insert PTX code into your CUDA program, so that you can mix CUDA/PTX in source code and still have benefits of code optimizations during the PTX generation. By rewriting parts of your code in PTX, you can 1) reduce total number of generated PTX instructions 2) exactly specify PTX instructions you need 3) tune the instructions through qualifiers 4) apply optimizations that are either lacking in the compiler or prohibited by C++ language extensions. Important! Using inline PTX Assembly will not make your code automatically faster than the one written in CUDA. It will only be faster if your hand-written PTX is better than the generated by the compiler.

In this implementation we will program some parts of the algorithm directly in PTX, so I highly recommend to check this short overview of inline ptx assembly if you have never used it before. The PTX instructions are well documented and can be found at PTX Instruction Set. We will now briefly review the PTX instructions used in this implementation.

4.1. Global Memory Loads

For global memory loads we will use ld.global.f32 instruction. Here, “ld” denotes “load” and “f32” - “32-bit float”. The following CUDA code

float reg; // single float register
float* gmem_ptr = data_in_global_memory; // pointer to global memory
reg = *gmem_ptr; // global memory -> register transfer

can be implemented in PTX as:

float reg; // single float register
float* gmem_ptr = data_in_global_memory; // pointer to global memory
asm volatile("ld.global.f32 %0, [%1];" : "=f"(reg) : "l"(gmem_ptr));

The f in "=f" denotes float datatype and the = modifier specifies that the register is written to. The l represents unsigned 64-bit integer. We also use volatile keyword to ensure that the instruction is not deleted or moved during generation of PTX.

4.2. Global Memory Stores

For global memory stores there is `st.global.f32 instruction:

float reg; // single float register
float* gmem_ptr = data_in_global_memory; // pointer to global memory
// *gmem_ptr = reg; can be implemented in PTX as:
asm volatile("st.global.f32 [%0], %1;" : : "l"(gmem_ptr), "f"(reg));

4.3. Global to Shared Memory Transfers

When you write something like this in CUDA:

__shared__ float smem_ptr[n]; // pointer to shared memory
float* gmem_ptr = data_in_global_memory; // pointer to global memory
*smem_ptr = *gmem_ptr; // global to shared memory transfer

a two-step process occurs. First, the data is fetched from global memory into registers and then that data is copied from registers into shared memory. Additionally the data is cached in all cache levels during the transfer.

Global to shared memory transfers

For this reason, a global to shared memory transfer in PTX consists of two data movement instructions ld.global and st.shared:

__shared__ float smem_ptr[n]; // pointer to shared memory
uint64_t smem_addr;
// convert generic address to shared address (store location for st.shared instruction)
asm volatile("cvta.to.shared.u64 %0, %1;" : "=l"(smem_addr) : "l"(smem_ptr));

float* gmem_ptr = data_in_global_memory; // pointer to global memory
float buffer;
// global memory -> register
asm volatile("ld.global.f32 %0, [%1];" : "=f"(buffer) : "l"(gmem_ptr));
// register -> shared memory
asm volatile("st.shared.f32 [%0], %1;" : : "l"(smem_addr), "f"(buffer));

Prior to Ampere architecture it was not possible to transfer data from global memory directly to shared memory mitigating storing in registers. Starting from Ampere architecture, there are asynchronous copy instructions that allow this. The usage of these instructions will be demonstrated later.

4.4. Vectorized Shared Memory Loads and Stores

In PTX you can also implement vectorized memory operations (loading/storing multiple elements with one instruction). Here, v4 denotes vector with four elements:

float reg0, reg1, reg2, reg3;
uint64_t addr;
...
// Shared memory 128-bit loads
asm volatile("ld.shared.v4.f32 {%0, %1, %2, %3}, [%4];"
             : "=f"(reg0), "=f"(reg1), "=f"(reg2), "=f"(reg3)
             : "l"(addr));
// Shared memory 128-bit stores
asm volatile("st.shared.v4.f32 [%0], {%1, %2, %3, %4};"
             :
             : "l"(addr), "f"(reg0), "f"(reg1), "f"(reg2), "f"(reg3));

4.5. Predicated Execution

In PTX conditional executions are implemented using optional guard predicates. The following CUDA code:

float reg;
float* ptr; //pointer to global memory
unsigned guard;
...
if (guard != 0) {
    *ptr = reg;
}

can be converted to PTX as:

float reg;
float* ptr;
unsigned guard;
...
asm volatile(".reg .pred p;\n\t" // declare predicate 'p'
             ".setp.ne.u32 p, %2, 0;\n\t" // set 'p' to true if (guard != 0); ne="not equal"
             "@p ld.global.f32 %0, [%1];\n\t" // execute instruction if 'p' is true
             : "=f"(reg)
             : "l"(ptr), "r"(guard));

We use guard predicates in combination with global load/store instructions to perform global memory access only if it is not out of bounds.

5. SGEMM Design

Let’s now break down the high-level design of the algorithm. The paper Strassen’s Algorithm Reloaded on GPUs contains, in my opinion, one of the best visualizations of the SGEMM design from the CUTLASS library. The SGEMM algorithm can be roughly divided into three main parts:

Each of these steps must be carefully optimized to achieve high overall performance. In the following sections, we’ll explore each step in detail and discuss efficient implementation strategies. It’s worth mentioning that the first step - “transferring data from global memory to shared memory” is the most challenging to grasp. However, once you understand this part, the remaining steps become much easier to follow.

5.1. Transferring data from global to shared memory

Source: Strassen’s Algorithm Reloaded on GPUs

To parallelize $C=AB$ on GPU, the matrix $C$ is partitioned into sub-matrices $\tilde{C}$ of size $m_S \times n_S$ and the sub-matrices are processed in parallel with one thread block computing one sub-matrix $\tilde{C}$ independently from other thread blocks. To compute $\tilde{C}$, we iterate over the dimension $K$. In each iteration, a submatrix $\tilde{A}$ of size $m_s \times k_s$ and a submatrix $\tilde{B}$ of size $k_s \times n_s$ are loaded from global into shared memory (see the figure above). These submatrices are then multiplied, and the result is used to update $\tilde{C}$ as $\tilde{C} += \tilde{A} \tilde{B}$. The sub-matrices $\tilde{A}, \tilde{B}, \tilde{C}$ are often called blocks or tiles. In total there are $K / k_s$ iterations (assuming the simplest case, where $K$ is divisible by $k_s$). The limited shared memory capacity is the reason why the dimension $K$ is divided into smaller $k_s$ blocks. Full $m_s \times K, K \times n_s$ blocks simply wouldn’t fit available shared memory. For now, don’t be distracted by why the matrices are loaded into shared memory and how exactly the matrices $\tilde{A}, \tilde{B}$ are multiplied, we will discuss it in the next chapter. Let’s focus on the efficient data movement from global to shared memory as our first step towards fast SGEMM.

The pseudo code of the algorithm, from the perspective of a thread block, is as follows:

// The shapes of block_a, block_b, block_c are (ms x ks), (ks x ns), (ms x ns)
// Each thread block computes one block of C:
block_c = 0
__shared__ float block_a[block_a_size]
__shared__ float block_b[block_b_size]
for (i=0; i<K/ks; i++) {
    block_a = load ith block of matrix A // from global into shared memory
    block_b = load ith block of matrix B // from global into shared memory
    block_c += block_a * block_b // compute matrix product and update block_c
}
store(block_c) // store to global memory

Data transfers from global memory to shared memory have significantly higher latency compared to arithmetic operations. During this time, threads are forced to stall, idly waiting for the data needed to compute block_a * block_b. One way to mitigate this latency is by overlapping data transfers with computations, leveraging instruction-level parallelism (ILP). In GEMM implementations, a technique known as double buffering is commonly used to achieve this overlap:

block_c = 0
// Shared Memory Double buffering
__shared__ float block_a[2][block_a_size] // 2x shared memory usage
__shared__ float block_b[2][block_b_size] // 2x shared memory usage
block_a[0] = load first block of matrix A
block_b[0] = load first block of matrix B

for (i=0; i<(K/ks-1); i++) {
    idx = i%2
    prefetch_idx = (i+1)%2
    // prefetch next blocks
    block_a[prefetch_idx] = load next block of matrix A
    block_a[prefetch_idx] = load next block of matrix B
    // use blocks loaded in previous iteration to calculate matrix product
    block_c += block_a[idx] * block_b[idx]
}
// final update of the accumulator using last blocks
block_c += block_a[prefetch_idx] * block_b[prefetch_idx]

store_to_global_memory(block_c)

Note that block_c += block_a[idx] * block_b[idx] doesn’t depend on blocks[prefetch_idx] allowing the arithmetic instructions to be issued in parallel with the data movement instructions. However, this comes at the cost of doubled shared memory usage, as we need to store two blocks instead of one. The good news is that modern GPUs have sufficient shared memory to support double-buffering.

We’ve already introduced several parameters such as block sizes $m_s, k_s, n_s$ and number of threads per thread block. The choice of these parameters highly depends on the shapes of the operands $A, B, C$, as well as the underlying GPU architecture. For example, cuBLAS implements multiple SGEMM kernels optimized for various matrix shapes and GPU architectures. At runtime, it selects the most appropriate kernel using a heuristic approach. The block sizes $m_s, k_s, n_s$ affect not only how the data will be fetched from global memory, but also how the work in all subsequent steps (shared memory loads, arithmetic operations, global memory stores) is organized among the threads to achieve the best possible performance. The choice of the block sizes and the number of threads per thread block also impact shared memory / register usage, which can result in decreased performance if not taken into account. As you might expect, identifying optimal parameter values requires excellent understanding of hardware and extensive experimentation. Fortunately, SGEMM is a well-studied problem and we can use the results from previous studies of cuBLAS and CUTLASS. For large square matrices (M=N=K > 1024) the combinations of $m_S \times n_S$ such as $128 \times 256$, $128 \times 128$ and $256 \times 128$ lead to optimal performance. From my tests, the configuration $m_s \times n_s \times k_s = 128 \times 128 \times 8$ with 256 threads per thread block achieved the highest TFLOP/S on my local RTX 3090 for matrix size problems 1024 <= M=N=K <= 2500. Therefore, we will start with implementation of a 128x128x8 SGEMM kernel. Now that we know the block dimensions and the number of threads per thread block, let’s discuss how to efficiently organize data loading from global memory and storage into shared memory.

First, we need to load 128x8 submatrix $\tilde{A}$ using 256 threads. This results in each thread loading 128*8/256 = 4 float elements from global memory. There are several different ways to organize the loading of the block. For global memory reads/stores you always want your accesses to be contiguous or coalesced, so that 32 threads in a wrap access 32 consecutive floats in memory. If a memory access is coalesced the minimum number of memory transactions will be used. However, it is not possible in case of the $\tilde{A}$ block: each row of the block contains only 8 consecutive elements. Nevertheless, even in such cases, consecutive threads in a wrap accessing consecutive elements in memory is preferable and usually results in better performance. The figure below shows how the loading of the block $\tilde{A}$ is implemented. Here, different colors represent different threads, whereas only first 16 threads are shown for simplicity. Four consecutive rows are loaded by 8 consecutive threads: the rows 1-4 are loaded by threads 0-7, the rows 5-8 are loaded by threads 8-15, the rows 9-12 are loaded by threads 16-23 and so on, with the last rows 125-128 are loaded by threads 248-255. We also transpose the block $\tilde{A}$ while storing in shared memory for better memory access pattern during the next computation step. Note how each thread stores 4 consecutive elements in shared memory. This allows us to use PTX vectorized stores st.shared.v4.f32.

Storing to shared memory using this naive scheme would result in shared memory bank conflicts. From the CUDA programming guide:

To achieve high bandwidth, shared memory is divided into equally-sized memory modules, called banks, which can be accessed simultaneously. Any memory read or write request made of n addresses that fall in n distinct memory banks can therefore be serviced simultaneously, yielding an overall bandwidth that is n times as high as the bandwidth of a single module.
However, if two addresses of a memory request fall in the same memory bank, there is a bank conflict and the access has to be serialized. The hardware splits a memory request with bank conflicts into as many separate conflict-free requests as necessary, decreasing throughput by a factor equal to the number of separate memory requests. If the number of separate memory requests is n, the initial memory request is said to cause n-way bank conflicts.

Shared memory has 32 banks that are organized such that successive 32-bit words map to successive banks. Imagine a float32 array of size 8x32 stored in row-major order as shown below.

In this context, colors and their shades represent memory banks: each row corresponds to 32 distinct memory banks, while each column represents a single memory bank. Here are two important notes about shared memory bank conflicts:

When you store (or load) 4 bytes(= 1 float) per thread, which is 4*32=128 bytes per warp, a CUDA device issues a single memory transaction (warp-wide) so that the shared memory access must be conflict-free across the whole wrap(=32 threads). In our case, we store 16 bytes(= 4 floats) per thread using the vector instructions. Warp-wide that will be a total of 512 bytes per request. The GPU splits the request into 4 memory transactions (threads 0-7 make up a transaction, threads 8-15 a transaction and so on), each of which is 128 byte wide. If we would store according to our scheme, then each thread within threads 0-7 would store to the same four columns (red color shades) or with other words to the same four memory banks causing bank conflicts. The same applies for other memory transactions i.e. threads 8-15, threads 16-23 and so on. One possible way to completely avoid bank conflicts would be to pad the leading dimension with 16 bytes (=4 floats) as shown below.

Now, if we store the data according to our scheme, each thread within threads 0-7 would accesses distinct memory banks, resulting in 32 memory banks being accessed per memory transaction. The same applies for the remaining memory transactions i.e. t8-t15, t16-t23 and so on. This is the reason why the leading dimension is 132 and not 128 in the implementation:

const int smem_a_ld = 132; // 128 + 4

To implement double-buffering and store two $\tilde{A}$ blocks, theoretically, we would need shared memory of size 2*132*8*4 bytes. However, we increase the size to the nearest power of 2 = 2*256*8*4 to enable fast switching. Compare the following code with the pseudocode presented at the beginning of the chapter:

// Double-buffering (blocks_b is omitted for simplicity)
__shared__ float __align__(2*256*8*sizeof(float)) blocks_a[2*256*8]
uint64_t lds_a_addr;
uint64_t sts_a_addr;
float* lds_a_ptr = blocks_a; // lds = load shared
float* sts_a_ptr = blocks_a; // sts = store shared
lds_a_addr = convert_to_addr(lds_a_ptr); // convert pointer to address for PTX load/store instructions
sts_a_addr = convert_to_addr(sts_a_ptr); // convert pointer to address for PTX load/store instructions

// store first block to first half of shared memory
sts_ptx(sts_a_addr);
// switch address to second half of shared memory
sts_a_addr ^= 8192;

for (int i=0; i<(K/ks-1); i++) {
    ...
    // store next block to second(first) half of shared memory
    sts_ptx(sts_a_addr);
    ...
    // load block from first(second) half of shared memory to compute c+=block_a*block_b
    lds_ptx(lds_a_addr);
    ...
    // swap the addresses for next iteration: lds_a_addr = sts_a_addr, sts_a_addr = lds_a_addr
    lds_a_addr ^= 8192;
    sts_a_addr ^= 8192;
    ...
}
...

First, we require blocks_a to be 2*256*8*4=2^14=16384-byte aligned. This implies the address of the first element of blocks_a to be divisible by 16384 or with other words the last 14 bits of the address are zero:

As each block size is 8192=2^13 bytes, switching between the blocks can now be implemented with just a single XOR instruction ^= 8192. The only drawback of this method is the unused shared memory (in this case 2*8*128*4 bytes). However, this can be ignored considering maximum amount of shared memory per thread block on modern GPUs.

Loading and storing a 8 x 128 submatrix $\tilde{B}$ is much simpler to manage due to its shape. Since the sub-matrix must not be transposed, the loading and storing schemes are identical:

We use 32 consecutive threads to load 32 consecutive elements, with each thread loading 4 elements, spaced apart by a stride of 32. Note that since we store data in 32 distinct shared memory banks, no padding is required, and bank conflicts are avoided. Furthermore, the block size 128*8 is naturally a power of two, eliminating the need for additional padding and allowing block switching with a single XOR ^=4096 instruction.

5.2. Shared Memory Loads and Arithmetic Operations

With blocks $\tilde{A}$ and $\tilde{B}$ now residing in shared memory, let’s discuss how to efficiently load from shared memory and compute block $\tilde{C}$. To do this, we’ll dive one level deeper into our parallelization strategy and describe the algorithm from a warp’s perspective:

Launched thread block consists of 256 threads, which corresponds to 256/32=8 warps. The block $\tilde{C}$, with dimensions $128 \times 128$, is, therefore, divided into 8 regions $\tilde{C}_W$ labeled $W1, …, W8$ in the figure. Each region $\tilde{C}_W$ has dimensions $m_W \times n_W = 32 \times 64$ and is computed by a single warp: $W1$ is computed by threads t0-t31, $W2$ is computed by threads t32-t63, and so on, with $W8$ computed by threads t224-t255. The figure above uses $W8$ as an example to demonstrate how a single $\tilde{C}_W$ region is computed. We iterate over the dimension $K$ and in each iteration we

As $k_S = 8$, there will be in total 8 iterations. This explanation is from the perspective of a warp. Now, let’s delve one final level deeper and examine how the work within a warp is distributed among its 32 threads.

Each thread in a wrap computes four 4x4 sub-matrices (=accumulators) within $\tilde{C}_W$ or if concatenated - 8x8 accumulator. To do this, each thread loads 8 elements from fragment_a, 8 elements from fragment_b (as illustrated for thread t0 in the figure), multiplies them and updates the accumulator using fused multiply-add (FMA) instructions. Since block_a was transposed in the previous step, the elements in fragment_a are stored contiguously in memory, allowing faster access through vectorized loads. The threads are arranged in a way that avoids bank conflicts and works around NVIDIA’s shared memory broadcast limitation. This limitation occurs when 4 floats loaded using 16-byte vector instruction must be broadcast to more than 4 consecutive threads within a warp.

Bringing everything together, the entire SGEMM algorithm can be visualized as follows:

As you might expect, the accumulators are frequently updated during the computation and need to be stored in the fastest memory - the register files. Each thread allocates float accumulator[8][8], so that the entire block $\tilde{C}$ of size $128 \times 128$ is stored in registers by the 256 threads. This works because 256=16*16, and the combined arrangement (16*8)x(16*8)=128x128 matches the size of $\tilde{C}$. Just as we used double buffering to load the blocks $\tilde{A}$ and $\tilde{B}$ (from global memory to shared memory), we now also double buffer the fragments to minimize memory transfer latencies when moving data from shared memory to registers. The pseudocode for the algorithm can be written as follows:

// Pseudocode
__shared__ float block_a[2][block_a_size]
__shared__ float block_b[2][block_b_size]
float fragment_a[2][8]
float fragment_b[2][8]
float accumulator[8][8]

block_a[0] = load first block of matrix A
block_b[0] = load first block of matrix B
fragment_a[0] = load first fragment from block_a[0]
fragment_b[0] = load first fragment from block_b[0]

for (block_k=0; block_k<(K/ks-1); block_k++) {
    block_idx = block_k % 2
    block_prefetch_idx = (block_k+1) % 2
    // prefetch next blocks (Shared Memory Double buffering)
    block_a[block_prefetch_idx] = load next block of matrix A
    block_a[block_prefetch_idx] = load next block of matrix B
    for (int warp_k=0; warp_k<8; warp_k++) {
        frag_idx = warp_k % 2
        frag_prefetch_idx = (warp_k + 1) % 2
        // prefetch next fragments (Register Double buffering)
        fragment_a[frag_prefetch_idx] = load next fragment from block_a[block_idx]
        fragment_b[frag_prefetch_idx] = load next fragment from block_b[block_idx]
        // use fragments loaded in previous iteration to calculate matrix product
        for (int i = 0; i < 8; i++) {
            for (int j = 0; j < 8; j++) {
                accumulator[i][j] += fragment_a[frag_idx][i] * fragment_b[frag_idx][j];
            }
        }
    }
    fragment_a[0] = load first fragment from block_a[block_prefetch_idx]
    fragment_b[0] = load first fragment from block_b[block_prefetch_idx]
}

// final update of the accumulator using last blocks
for (int warp_k=0; warp_k<8; warp_k++) {
    frag_idx = warp_k % 2
    frag_prefetch_idx = (warp_k + 1) % 2
    // prefetch next fragments (Register Double buffering)
    fragment_a[frag_prefetch_idx] = load next fragment from block_a[block_prefetch_idx]
    fragment_b[frag_prefetch_idx] = load next fragment from block_b[block_prefetch_idx]
    // use fragments loaded in previous iteration to calculate matrix product
    for (int i = 0; i < 8; i++) {
        for (int j = 0; j < 8; j++) {
            accumulator[i][j] += fragment_a[frag_idx][i] * fragment_b[frag_idx][j];
        }
    }
}

// After completing the matrix multiplication C=A*B, we perform one final update to the accumulator
// to compute  C=alpha*A*B before storing the result back to global memory:
for (int i=0; i<8; i++) {
    for (int j=0; j<8; j++) {
        accumulator[i][j] *= alpha;
    }
}

store_to_global_memory(accumulator)

5.3. Coalesced Global Memory Stores Through Shared Memory

Just as with global memory reads, we want our global memory writes to be coalesced. However, directly storing the accumulators to global memory based on our current mapping

would result in random memory accesses, significantly hurting performance. To fix this, we use shared memory as a buffer to rearrange the accumulators, enabling coalesced global memory writes. At this stage, the accumulators have already been computed, so we no longer need shared memory for computation. Transferring data from registers to shared memory is fast. The overhead of these additional transfers from registers to shared memory is negligible compared to the performance gains achieved through coalesced writes. We write the accumulator’s elements to shared memory row by row according to the following scheme:

The first row, containing 32 elements, is copied to the first 32 consecutive memory addresses in shared memory. Similarly, the second row is copied to the next 32 consecutive memory addresses, and so on with all 16 rows have been copied to shared memory. Next, we iterate through the rows in shared memory, and in each iteration, we store a row (containing 32 elements) to global memory using coalesced writes:

The process is then repeated for the other three 4x4 accumulators of the threads.

To compute $C := \alpha AB + \beta C$, we make a slight adjustment to the process of storing the data to global memory. After copying the accumulator from registers to shared memory, we check if beta != 0.0. If true, we load (using coalesced loads) the corresponding element from global memory into a register, multiply it by beta and add the result to the accumulator stored in shared memory. Finally, we store the updated accumulator alpha*A*B+beta*C from shared memory to global memory using coalesced writes.

6. Performance Analysis

So far, we have discussed the design of the 128x128x8 SGEMM kernel. Its implementation is available at 128x128x8.cuh and closely follows the pseudo-code outlined earlier. Let’s now benchmark this kernel to evaluate its performance. First, we conduct a benchmark with locked clock frequencies:

The benchmark results show that the implementation outperforms cuBLAS when clock speeds remain constant. However, performance alone is not enough; we also need to consider power consumption. To evaluate both metrics, we run the benchmark with unlocked clock frequencies:

This reveals the effect of throttling due to reaching power limits. While the 128x128x8 kernel is, on average, 3–4% faster than cuBLAS, it consumes 12% more power. The increased power consumption causes the GPU to operate near the power limit for matrix sizes m=n=k>4000, resulting in reduced clock speeds and overall performance degradation. This the reason why optimizing both runtime and power consumption is required for achieving a balanced and efficient implementation.

We can slightly improve the runtime of the kernel by utilizing vectorized global texture loads. The new kernel is available at 128x128x8_texld. Since the vectorized load instructions impose alignment constraints on the input data, we first verify the memory alignment and ensure the leading dimensions of matrices A and B are divisible by 4:

bool is_aligned = (((unsigned)lda & 3u) == 0) && (((unsigned)ldb & 3u) == 0)
                    && (((unsigned long)A & 15u) == 0) && (((unsigned long)B & 15u) == 0);

If the input data is aligned, we can use the vectorized load instructions. First we need to create texture objects, texture descriptors, and resource descriptors. These are configured to handle float data type with four 32-bit channels (x, y, z, w). The texture objects are then bound to the operands A, B, and passed to the kernel instead of raw pointers A, B.

cudaResourceDesc resDesc;
cudaTextureDesc texDesc;
cudaTextureObject_t tex_a = 0;
cudaTextureObject_t tex_b = 0;
...
if (is_aligned) {
    memset(&texDesc, 0, sizeof(texDesc));
    texDesc.readMode = cudaReadModeElementType;
    texDesc.normalizedCoords = 0;
    memset(&resDesc, 0, sizeof(resDesc));
    resDesc.resType = cudaResourceTypeLinear;
    resDesc.res.linear.desc.f = cudaChannelFormatKindFloat;
    resDesc.res.linear.desc.x = 32;
    resDesc.res.linear.desc.y = 32;
    resDesc.res.linear.desc.z = 32;
    resDesc.res.linear.desc.w = 32;
    resDesc.res.linear.devPtr = A;
    resDesc.res.linear.sizeInBytes = m * lda * sizeof(float);
    cudaCreateTextureObject(&tex_a, &resDesc, &texDesc, NULL);
    resDesc.res.linear.devPtr = B;
    resDesc.res.linear.sizeInBytes = k * ldb * sizeof(float);
    cudaCreateTextureObject(&tex_b, &resDesc, &texDesc, NULL);
    sgemm_texld_128x128x8<<<grid, threads>>>(m,
                                             n,
                                             k,
                                             *alpha,
                                             tex_a,
                                             lda,
                                             tex_b,
                                             ldb,
                                             *beta,
                                             C,
                                             ldc);
    cudaDestroyTextureObject(tex_a);
    cudaDestroyTextureObject(tex_b);
}

Within the kernel, we load data through texture objects using the tex1Dfetch function, which compiles to a single PTX instruction:

float4 texld_a_buffer;
texld_a_buffer = tex1Dfetch<float4>(tex_a, texld_a_offset);

We use global texture loads over normal vectorized global loads (ld.global.v4.f32) because texture loads handle out-of-bounds reads gracefully by returning zeros, avoiding the need for predicated execution. This simplification leads to more efficient code:

Lastly, we developed an 128x256x8 SGEMM kernel leveraging bigger block size $n_S=256$ and asynchronous copy instructions (cp.async.ca.shared.global) which are supported starting with the Ampere architecture. The main advantage of these instructions is that one can overlay computation with memory transfers and avoid pipeline stalls. Additionally, they allow to copy data from global memory directly into shared memory bypassing registers:

By simply replacing the normal global load instructions with cp.async in the 128x128x8 kernel results in degraded performance - possibly due to higher latencies of the cp.async instructions or suboptimal compiler optimizations. However, combining increased block size with cp.async yields superior results in both speed and power efficiency:

Our final implementation combines the 128x128x8 and 128x256x8 kernels. For smaller matrices m=n < 2500, we use the 128x128x8 kernel, otherwise, the 128x256x8 kernel.

making AI inference run really fast.


Harshit Kumar

Matrix Multiplication in CUDA

Matrix multiplication is at the heart of deep learning. In this evolving world of LLMs, the need for fast and efficient matrix multiplications is paramount. Nvidia CUDA allows you to perform matrix operations on GPU in a faster way.

CUDA (Compute Unified Device Architecture) is a parallel computing platform and application programming interface (API) model. CUDA programming model provides an abstraction of GPU architecture (API for GPUs).

In this blog post, we will explore how to implement matrix multiplication using CUDA. We will start with a naive implementation on the CPU and then demonstrate how to significantly speed up the process using CUDA.

Naive C++ Implementation on CPU

Since in most hardwares, matrices are stored in row-major format, let’s define our 2d matrices as row-major 1d arrays.

struct Matrix
{
    int height;
    int width;
    float *elements; // height x width
    // you can also use std::vector<float> elements for automatic memory management
};

Matrix multiplication for computing each element of matrix C from matrices A and B can be written as follows:

where i and j are the row and column indices of the resulting matrix C and k is the index used for the summation over the common dimension.

Our naive matrix multplication in C++ on CPU is:

void matMulCPU(const Matrix &A, const Matrix &B, Matrix &C)
{
    for (int row = 0; row < A.height; ++row)
    {
        for (int col = 0; col < B.width; ++col)
        {
            float cValue = 0;
            // C[i][j] = sum_k A[i][k] * B[k][j]
            for (int k = 0; k < A.width; ++k)
                cValue += A.elements[row * A.width + k] * B.elements[k * B.width + col];
            C.elements[row * C.width + col] = cValue;
        }
    }
}

We can use the below main() function to call our matMulCPU() and measure its performance.

// Function to initialize a matrix with random values
void initializeMatrix(Matrix &mat)
{
    for (int i = 0; i < mat.height * mat.width; ++i)
        mat.elements[i] = static_cast<float>(rand() % 100);
}

int main()
{
    int M = 1024; // Rows of A and C
    int K = 768; // Columns of A and rows of B
    int N = 1024; // Columns of B and C

    // Allocate matrices A, B, and C
    Matrix A = {M, K, new float[M * K]}; // 1024x768
    Matrix B = {K, N, new float[K * N]}; // 768x1024
    Matrix C = {M, N, new float[M * N]}; // 1024x1024

    // Initialize matrices A and B with random values
    initializeMatrix(A);
    initializeMatrix(B);

    // Measure the time taken for matrix multiplication on the CPU
    auto start = std::chrono::high_resolution_clock::now();
    matMulCPU(A, B, C);
    auto stop = std::chrono::high_resolution_clock::now();

    std::chrono::duration<float> duration = stop - start;
    cout << "CPU matrix multiplication time: " << duration.count() * 1000.0f << " ms" << endl;

    // Clean up memory
    delete[] A.elements;
    delete[] B.elements;
    delete[] C.elements;

    return 0;
}

Naive CUDA Kernel

In CUDA, we define a CUDA kernel, which is a function (e.g. C++ function) executed by CUDA.

In CUDA programming model, there is a three-level hierarchy. The threads are the smallest unit of execution. These threads are grouped into a CUDA thread block. CUDA blocks are grouped into arrays called grids. The kernel is written from the perspective of a single thread in CUDA. Thus, a kernel is executed as a grid of blocks of threads.

On a CPU, matrix multiplication is typically performed sequentially, where each element of the output matrix is computed one after another. This process can be slow for large matrices due to the limited number of CPU cores available for parallel execution. In contrast, the GPU excels at parallel processing. A CUDA kernel is executed by many threads running simultaneously, allowing for significant speedup in computations like matrix multiplication. The GPU’s architecture enables it to handle thousands of threads concurrently, making it well-suited for tasks with high levels of parallelism.

Let’s re-write the above matrix multiplication code in CUDA. We use __global__ keyword to define a CUDA kernel.  Here, we assign a thread for calculation of each element of output matrix C. And, multiple such threads are run in parallel. Each thread reads one row of A and one column of B to compute one element of C.

Threads and blocks are indexed using the built-in 3D variable threadIdx and blockIdx. The blockDim gives the dimension of thread block. We can access index using dot attribute e.g. threadIdx.x, threadIdx.y, and threadIdx.z. Thus, for 2d thread block, we can access particular element of C using a combination of these as shown in below code.

__global__ void matMulNaiveKernel(Matrix A, Matrix B, Matrix C)
{
    int row = blockIdx.y * blockDim.y + threadIdx.y;
    int col = blockIdx.x * blockDim.x + threadIdx.x;

    // Each thread accumulates one element of C by accumulating results into cValue
    float cValue = 0;

    // C[i][j] = sum_k A[i][k] * B[k][j]
    // Iterates over common dimensions of A and B (k = A.width = B.height)
    if (row < A.height && col < B.width)
    {
        for (int k = 0; k < A.width; ++k)
            cValue += A.elements[row * A.width + k] * B.elements[k * B.width + col];
        C.elements[row * C.width + col] = cValue;
    }
}

We create a 16x16 thread block (256 threads with 16 each in x and y-direction). We define (B.width/BLOCK_SIZE, A.height/BLOCK_SIZE) blocks per grid. Extra operations below is to take care of the last tile if size isn’t perfectly divisible.

#define BLOCK_SIZE 16
dim3 threadsPerBlock(BLOCK_SIZE, BLOCK_SIZE);
dim3 blocksPerGrid((B.width + threadsPerBlock.x - 1) / threadsPerBlock.x,
                (A.height + threadsPerBlock.y - 1) / threadsPerBlock.y);
runKernel(matMulNaiveKernel, A, B, C, blocksPerGrid, threadsPerBlock);

This kernel is called with device (gpu) matrices A, B, and C as follows:

kernel<<<blocksPerGrid, threadsPerBlock>>>(d_A, d_B, d_C);

This setup ensures that the CUDA kernel efficiently processes the entire matrix by dividing the workload among the available threads and blocks.

To execute CUDA program:

We pass our kernel to the runKernel() function that also takes CPU matrices A and B. It copies the data from CPU to GPU, runs kernel, copy result from GPU to CPU, and return the result matrix C.

void runKernel(void(*kernel)(Matrix, Matrix, Matrix),
               const Matrix &A, const Matrix &B, Matrix &C,
               dim3 gridDim, dim3 blockDim)
{
    // Load matrices to device memory
    Matrix d_A, d_B, d_C;
    size_t size_A = A.width * A.height * sizeof(float);
    size_t size_B = B.width * B.height * sizeof(float);
    size_t size_C = C.width * C.height * sizeof(float);
    d_A.width = A.width; d_A.height = A.height;
    d_B.width = B.width; d_B.height = B.height;
    d_C.width = C.width; d_C.height = C.height;

    // Allocate device memory
    CUDA_CHECK_ERROR(cudaMalloc(&d_A.elements, size_A));
    CUDA_CHECK_ERROR(cudaMalloc(&d_B.elements, size_B));
    CUDA_CHECK_ERROR(cudaMalloc(&d_C.elements, size_C));

    // Copy A, B to device memory
    CUDA_CHECK_ERROR(cudaMemcpy(d_A.elements, A.elements, size_A, cudaMemcpyHostToDevice));
    CUDA_CHECK_ERROR(cudaMemcpy(d_B.elements, B.elements, size_B, cudaMemcpyHostToDevice));

    auto start = std::chrono::high_resolution_clock::now();

    // Launch kernel
    kernel<<<gridDim, blockDim>>>(d_A, d_B, d_C);

    // Synchronize device memory
    CUDA_CHECK_ERROR(cudaDeviceSynchronize());

    auto end = std::chrono::high_resolution_clock::now();
    std::chrono::duration<float> duration = end - start;
    std::cout << "Kernel execution time: " << duration.count() * 1000.0f << " ms" << std::endl;

    // Copy C from device memory to host memory
    CUDA_CHECK_ERROR(cudaMemcpy(C.elements, d_C.elements, size_C, cudaMemcpyDeviceToHost));

    // Free device memory
    CUDA_CHECK_ERROR(cudaFree(d_A.elements));
    CUDA_CHECK_ERROR(cudaFree(d_B.elements));
    CUDA_CHECK_ERROR(cudaFree(d_C.elements));
}

And, we call runKernel() function in above defined main() function.

CUDA Shared Memory Kernel

The previous CUDA kernel uses DRAM, but we can optimize performance by leveraging the GPU’s shared memory. Shared memory is faster but has limited capacity, so we cannot load entire matrices at once. Instead, we divide the matrices into smaller sub-matrices, or tiles, that fit into shared memory.

Shared memory is allocated per thread block, allowing threads within the same block to communicate efficiently. Each thread block is responsible for computing one square sub-matrix \(C_{sub}\) of C by loading tiles of input matrices A and B from global memory to shared memory. Each thread within the block computes a single element of \(C_{sub}\) by iterating over the corresponding elements in the shared memory tiles, accumulating the results of the products. Finally, each thread writes its computed value to the appropriate position in global memory.

#define TILE_SIZE 16

// Kernel for matrix multiplication using tiling and shared memory
__global__ void matMulSharedMemoryKernel(Matrix A, Matrix B, Matrix C)
{
    // Shared memory for tiles of A and B
    __shared__ float shared_A[TILE_SIZE][TILE_SIZE];
    __shared__ float shared_B[TILE_SIZE][TILE_SIZE];

    // Calculate the global row and column index of the element
    int globalRow = blockIdx.y * blockDim.y + threadIdx.y;
    int globalCol = blockIdx.x * blockDim.x + threadIdx.x;

    float Cvalue = 0.0f;

    // Thread row and column within Csub
    int row = threadIdx.y;
    int col = threadIdx.x;

    // Loop over the tiles of the input matrices
    // A.width/TILE_SIZE and B.height/TILE_SIZE; take care of the last tile
    for (int m = 0; m < (A.width + TILE_SIZE - 1) / TILE_SIZE; ++m)
    {
        // Load elements of A into shared memory
        // if shared memory defined using 1d array, we'd have used shared_A[row * TILE_SIZE + col]
        if (row < A.height && (m * TILE_SIZE + col) < A.width)
        {
            shared_A[row][col] = A.elements[globalRow * A.width + m * TILE_SIZE + col];
        } else
        {
            // When matrix dimensions are not exact multiples of the tile size,
            // some threads in the last blocks might access elements outside
            // the matrix boundaries. By setting out-of-bounds elements to zero,
            // we ensure that these threads do not contribute invalid values to final result.
            // e.g. Matrix A = [100x100] and TILE_SIZE = 16
            shared_A[row][col] = 0.0f;
        }
        // Load elements of B into shared memory
        if (col < B.width && (m * TILE_SIZE + row) < B.height)
        {
            shared_B[row][col] = B.elements[(m * TILE_SIZE + row) * B.width + globalCol];
        } else
        {
            shared_B[row][col] = 0.0f;
        }
        // Synchronize to ensure all threads have loaded their elements
        __syncthreads();

        // Compute the partial result
        for (int k = 0; k < TILE_SIZE; ++k)
            Cvalue += shared_A[row][k] * shared_B[k][col];

        // Synchronize to ensure all threads have completed the computation
        __syncthreads();
    }

    // Write the result to global memory
    if (globalRow < C.height && globalCol < C.width)
        C.elements[globalRow * C.width + globalCol] = Cvalue;
}

We can call our kernel as follows:

dim3 blockDim(TILE_SIZE, TILE_SIZE);
dim3 gridDim((C.width + TILE_SIZE - 1) / TILE_SIZE, (C.height + TILE_SIZE - 1) / TILE_SIZE);
runKernel(matMulSharedMemoryKernel, A, B, C, gridDim, blockDim);

CUDA Matrix Multiplication Comparison

The kernel execution time of above kernels on Tesla T4 on google colab is as follows.

The CUDA parallelism significantly improves the CPU computation time. The shared memory kernel achieves the fastest execution time.

The full code is availble at https://github.com/kHarshit/cuda-programming

Further Optimization

There are other ways to optimize the CUDA matrix multplication kernel further, such as:

By combining these advanced optimization techniques with shared memory, you can achieve even greater performance gains for matrix multiplication on CUDA-enabled GPUs.

References

You May Also Like


January 2009

Optimizing Matrix
Transpose in CUDA
Greg Ruetsch

gruetsch@nvidia.com

Paulius Micikevicius

pauliusm@nvidia.com

  ii

January 2009
  3
 Chapter 1.
Introduction
Optimizing CUDA Memory
Management in Matrix
Transpose
This document discusses aspects of CUDA application performance related to
efficient use of GPU memories and data management as applied to a matrix
transpose.  In particular, this document discusses the following issues of memory
usage:
 coalescing data transfers to and from global memory
 shared memory bank conflicts
 partition camping
There are other aspects of efficient memory usage not discussed here, such as
data transfers between host and device, as well as constant and texture memories.
Both coalescing and partition camping deal with data transfers between global
device and on-chip memories, while shared memory bank conflicts deal with on-
chip shared memory.
The reader should be familiar with basic CUDA programming concepts such as
kernels, threads, and blocks, as well as a basic understanding of the different
memory spaces accessible by CUDA threads.  A good introduction to CUDA
programming is given in the CUDA Programming Guide as well as other
resources on CUDA Zone (http://www.nvidia.com/cuda).
The matrix transpose problem statement is given next, followed by a brief
discussion of performance metrics, after which the remainder of the document
presents a sequence of CUDA matrix transpose kernels which progressively
address various performance bottlenecks.

Matrix Transpose Characteristics
In this document we optimize a transpose of a matrix of floats that operates out-
of-place, i.e. the input and output matrices address separate memory locations.
For simplicity and brevity in presentation, we consider only square matrices
whose dimensions are integral multiples of 32 on a side, the tile size, through the
Optimizing Matrix Transpose in CUDA

4
January 2009
document. However, modifications of code required to accommodate matrices of
arbitrary size are straightforward.

Code Highlights and Performance Measurements
The host code for all the transpose cases is given in Appendix A.  The host code
performs typical tasks: data allocation and transfer between host and device, the
launching and timing of several kernels, result validation, and the deallocation of
host and device memory.
In addition to different matrix transposes, we run kernels that execute matrix
copies.  The performance of the matrix copies serve as benchmarks that we
would like the matrix transpose to achieve.
For both the matrix copy and transpose, the relevant performance metric is the
effective bandwidth, calculated in GB/s as twice the size of the matrix – once for
reading the matrix and once for writing – divided by the time of execution.  Since
timing is performed in loops executed NUM_REPS times, which is defined at the
top of the code, the effective bandwidth is also normalized by NUM_REPS.
Looping NUM_REPS times over code for measurement is done in two different
fashions: looping over kernel launches, and looping within the kernel over the
load and stores.  The host code for these measurements is given below:
// take measurements for loop over kernel launches
    cudaEventRecord(start, 0);
    for (int i=0; i < NUM_REPS; i++) {
      kernel<<<grid, threads>>>(d_odata, d_idata,size_x,size_y,1);
    }
    cudaEventRecord(stop, 0);
    cudaEventSynchronize(stop);
    float outerTime;
    cudaEventElapsedTime(&outerTime, start, stop);

    ...

 // take measurements for loop inside kernel
    cudaEventRecord(start, 0);
    kernel<<<grid,threads>>>
        (d_odata, d_idata, size_x, size_y, NUM_REPS);
    cudaEventRecord(stop, 0);
    cudaEventSynchronize(stop);
    float innerTime;
    cudaEventElapsedTime(&innerTime, start, stop);

The first timing is done by a for loop in the host code, the second by passing
NUM_REPS as a parameter to the kernel.  A simple copy kernel is shown below:

__global__ void copy(float *odata, float* idata, int width,
                                     int height, int nreps)
{
  int xIndex = blockIdx.x*TILE_DIM + threadIdx.x;
  int yIndex = blockIdx.y*TILE_DIM + threadIdx.y;

  int index  = xIndex + width*yIndex;
Optimizing Matrix Transpose in CUDA

January 2009
  5
  for (int r=0; r < nreps; r++) {
    for (int i=0; i<TILE_DIM; i+=BLOCK_ROWS) {
      odata[index+i*width] = idata[index+i*width];
    }
  }
}

The difference between these two timings is the overhead of the kernel launch,
which should be consistent between different kernels, as well as the time spent in
calculating the matrix indices at the beginning of each kernel.  In addition,
looping over kernel launches also acts as a synchronization mechanism.  When
the kernel is launched multiple times from a loop in host code, all blocks from
one kernel launch must complete execution before any block of a following
launch can begin.  As a result the set of active blocks and hence memory access
patterns resets every loop iteration.  When the loop is performed within the
kernels, the set of active thread blocks has more opportunity to diverge as
execution progresses through the timing loop.
Both methods of timing code provide useful measurements, the first indicating
what one would typically use as an overall performance metric, and the second as
a means of comparing the data movement times between kernels.
In the following section we present different kernels called from the host code,
each addressing different performance issues.  All kernels in this study launch
thread blocks of dimension 32x8, where each block transposes (or copies) a tile
of dimension 32x32.  As such, the parameters TILE_DIM and BLOCK_ROWS
are set to 32 and 8, respectively.  Using a thread block with fewer threads than
elements in a tile is advantageous for the matrix transpose in that each thread
transposes several matrix elements, four in our case, and much of the cost of
calculating the indices is amortized over these elements.

January 2009
  6
2. Copy and Transpose Kernels
Simple copy
The first two cases we consider are a naïve transpose and simple copy, each
using blocks of 32x8 threads on a 32x32 matrix tiles.  The copy kernel was given
in the previous section, and shows the basic layout for all of the kernels.  The
first two arguments odata and idata are pointers to the input and output
matrices, width and height are the matrix x and y dimensions, and nreps
determines how many times the loop over data movement between matrices is
performed.  In this kernel, the global 2D matrix indices xIndex and yIndex
are calculated, which are in turn used to calculate index, the 1D index used by
each thread to access matrix elements.   The loop over i adds additional offsets
to index so that each thread copies multiple elements of the array, and the loop
over r is used for timing the data transfer from input to output array multiple
times.
Naïve transpose
The naïve transpose:
__global__ void transposeNaive(float *odata, float* idata,
                         int width, int height, int nreps)
{
  int xIndex = blockIdx.x*TILE_DIM + threadIdx.x;
  int yIndex = blockIdx.y*TILE_DIM + threadIdx.y;

  int index_in  = xIndex + width * yIndex;
  int index_out = yIndex + height * xIndex;
  for (int r=0; r < nreps; r++) {
    for (int i=0; i<TILE_DIM; i+=BLOCK_ROWS) {
      odata[index_out+i] = idata[index_in+i*width];
    }
  }
}
is nearly identical to the copy kernel above, with the exception that  index, the
array index used to access elements in both input and output arrays for the copy
kernel, is replaced by the two indices index_in (equivalent to index in the
copy kernel), and index_out.  Each thread executing the kernel transposes
four elements from one column of the input matrix to their transposed locations
in one row of the output matrix.
Optimizing Matrix Transpose in CUDA

January 2009
  7
The performance of these two kernels on a 2048x2048 matrix using a GTX280 is
given in the following table:

Effective Bandwidth (GB/s)
2048x2048, GTX 280

Loop over kernel
Loop in kernel
Simple Copy
96.9
81.6
Naïve Transpose
2.2
2.2

The minor differences in code between the copy and naïve transpose kernels have
a profound effect on performance - nearly two orders of magnitude.  This brings
us to our first optimization technique: global memory coalescing.
Coalesced Transpose
Because device memory has a much higher latency and lower bandwidth than on-
chip memory, special attention must be paid to how global memory accesses are
performed, in our case loading data from idata and storing data in odata.  All
global memory accesses by a half-warp of threads can be coalesced into one or
two transactions if certain criteria are met.  These criteria depend on the compute
capability of the device, which can be determined, for example, by running the
deviceQuery SDK example.  For compute capabilities of 1.0 and 1.1, the
following conditions are required for coalescing:
 threads must access either 32- 64-, or 128-bit words, resulting in either one
transaction (for 32- and 64-bit words) or two transactions (for 128-bit words)
 All 16 words must lie in the same aligned segment of 64 or 128 bytes for 32-
and 64-bit words, and for 128-bit words the data must lie in two contiguous
128 byte aligned segments
 The threads need to access words in sequence.  If the k-th thread is to access
a word, it must access the k-th word, although not all threads need to
participate.
For devices with compute capabilities of 1.2, requirements for coalescing are
relaxed.  Coalescing into a single transaction can occur when data lies in 32-, 64-,
and 128-byte aligned segments, regardless of the access pattern by threads within
the segment.  In general, if a half-warp of threads access N segments of memory,
N memory transactions are issued.
In a nutshell, if a memory access coalesces on a device of compute capability 1.0
or 1.1, then it will coalesce on a device of compute capability 1.2 and higher.  If
it doesn’t coalesce on a device of compute capability 1.0 or 1.1, then it may
Optimizing Matrix Transpose in CUDA

8
January 2009
either completely coalesce or perhaps result in a reduced number of memory
transactions, on a device of compute capability 1.2 or higher.
For both the simple copy and naïve transpose, all loads from idata  coalesce on
devices with any of the compute capabilities discussed above.  For each iteration
within the i-loop, each half warp reads 16 contiguous 32-bit words, or one half
of a row of a tile.  Allocating device memory through cudaMalloc() and
choosing TILE_DIM to be a multiple of 16 ensures alignment with a segment of
memory, therefore all loads are coalesced.
Coalescing behavior differs between the simple copy and naïve transpose kernels
when writing to odata.  For the simple copy, during each iteration of the i-
loop, a half warp writes one half of a row of a tile in a coalesced manner.  In the
case of the naïve transpose, for each iteration of the i-loop a half warp writes one
half of a column of floats to different segments of memory, resulting in 16
separate memory transactions, regardless of the compute capability.
The way to avoid uncoalesced global memory access is to read the data into
shared memory, and have each half warp access noncontiguous locations in
shared memory in order to write contiguous data to odata.  There is no
performance penalty for noncontiguous access patters in shared memory as there
is in global memory, however the above procedure requires that each element in
a tile be accessed by different threads, so a __synchthreads() call is
required to ensure that all reads from idata to shared memory have completed
before writes from shared memory to odata commence.  A coalesced transpose
is listed below:
__global__ void transposeCoalesced(float *odata,
            float *idata, int width, int height, int nreps)
{
  __shared__ float tile[TILE_DIM][TILE_DIM];

  int xIndex = blockIdx.x*TILE_DIM + threadIdx.x;
  int yIndex = blockIdx.y*TILE_DIM + threadIdx.y;
  int index_in = xIndex + (yIndex)*width;

  xIndex = blockIdx.y * TILE_DIM + threadIdx.x;
  yIndex = blockIdx.x * TILE_DIM + threadIdx.y;
  int index_out = xIndex + (yIndex)*height;

  for (int r=0; r < nreps; r++) {
    for (int i=0; i<TILE_DIM; i+=BLOCK_ROWS) {
      tile[threadIdx.y+i][threadIdx.x] =
        idata[index_in+i*width];
    }

    __syncthreads();

    for (int i=0; i<TILE_DIM; i+=BLOCK_ROWS) {
      odata[index_out+i*height] =
        tile[threadIdx.x][threadIdx.y+i];
    }
  }
}
A depiction of the data flow of a half warp in the coalesced transpose kernel is
given below.  The half warp writes four half rows of the idata matrix tile to the
Optimizing Matrix Transpose in CUDA

January 2009
  9
shared memory 32x32 array “tile” indicated by the yellow line segments.
After a __syncthreads() call to ensure all writes to tile are completed,
the half warp writes four half columns of tile to four half rows of an odata
matrix tile, indicated by the green line segments.

With the improved access pattern to memory in odata, the writes are coalesced
and we see an improved performance:

Effective Bandwidth (GB/s)
2048x2048, GTX 280

Loop over kernel
Loop in kernel
Simple Copy
96.9
81.6
Naïve Transpose
2.2
2.2
Coalesced Transpose
16.5
17.1

While there is a dramatic increase in effective bandwidth of the coalesced
transpose over the naïve transpose, there still remains a large performance gap
between the coalesced transpose and the copy.  The additional indexing required
by the transpose doesn’t appear to be the cause for the performance gap, since the
results in the “Loop in kernel” column, where the index calculation is amortized
over 100 iterations of the data movement, also shows a large performance
difference.  One possible cause of this performance gap is the synchronization
barrier required in the coalesced transpose.  This can be easily assessed using the
following copy kernel which utilizes shared memory and contains a
__syncthreads() call:
__global__ void copySharedMem(float *odata, float *idata,
                          int width, int height, int nreps)
{
idata
odata
tile
Optimizing Matrix Transpose in CUDA

10
January 2009
  __shared__ float tile[TILE_DIM][TILE_DIM];

  int xIndex = blockIdx.x*TILE_DIM + threadIdx.x;
  int yIndex = blockIdx.y*TILE_DIM + threadIdx.y;

  int index  = xIndex + width*yIndex;
  for (int r=0; r < nreps; r++) {
    for (int i=0; i<TILE_DIM; i+=BLOCK_ROWS) {
      tile[threadIdx.y+i][threadIdx.x] =
        idata[index+i*width];
    }

    __syncthreads();

    for (int i=0; i<TILE_DIM; i+=BLOCK_ROWS) {
      odata[index+i*width] =
        tile[threadIdx.y+i][threadIdx.x];
    }
  }
}

The __syncthreads() call is not needed for successful execution of this
kernel, as threads do not share data, and is included only to assess the cost of the
synchronization barrier in the coalesced transpose.  The results are shown in the
following modified table:

Effective Bandwidth (GB/s)
2048x2048, GTX 280

Loop over kernel
Loop in kernel
Simple Copy
96.9
81.6
Shared Memory Copy
80.9
81.1
Naïve Transpose
2.2
2.2
Coalesced Transpose
16.5
17.1

The shared memory copy results seem to suggest that the use of shared memory
with a synchronization barrier has little effect on the performance, certainly as far
as the “Loop in kernel” column indicates when comparing the simple copy and
shared memory copy.  When comparing the coalesced transpose and shared
memory copy kernels, however, there is one performance bottleneck regarding
how shared memory is accessed that needs to be addressed: shared memory bank
conflicts.
Optimizing Matrix Transpose in CUDA

January 2009
  11
Shared memory bank conflicts
Shared memory is divided into 16 equally-sized memory modules, called banks,
which are organized such that successive 32-bit words are assigned to successive
banks.   These banks can be accessed simultaneously, and to achieve maximum
bandwidth to and from shared memory the threads in a half warp should access
shared memory associated with different banks.  The exception to this rule is
when all threads in a half warp read the same shared memory address, which
results in a broadcast where the data at that address is sent to all threads of the
half warp in one transaction.
One can use the warp_serialize flag when profiling CUDA applications to
determine whether shared memory bank conflicts occur in any kernel.  In
general, this flag also reflects use of atomics and constant memory, however
neither of these are present in our example.
The coalesced transpose uses a 32x32 shared memory array of floats.  For this
sized array, all data in columns k and k+16 are mapped to the same bank.  As a
result, when writing partial columns from tile in shared memory to rows in
odata the half warp experiences a 16-way bank conflict and serializes the
request.  A simple to avoid this conflict is to pad the shared memory array by one
column:
  __shared__ float tile[TILE_DIM][TILE_DIM+1];
The padding does not affect shared memory bank access pattern when writing a
half warp to shared memory, which remains conflict free, but by adding a single
column now the access of a half warp of data in a column is also conflict free.
The performance of the kernel, now coalesced and memory bank conflict free, is
added to our table below:

Effective Bandwidth (GB/s)
2048x2048, GTX 280

Loop over kernel
Loop in kernel
Simple Copy
96.9
81.6
Shared Memory Copy
80.9
81.1
Naïve Transpose
2.2
2.2
Coalesced Transpose
16.5
17.1
Bank Conflict Free Transpose
16.6
17.2

While padding the shared memory array did eliminate shared memory bank
conflicts, as was confirmed by checking the warp_serialize flag with the
CUDA profiler, it has little effect (when implemented at this stage) on
performance.  As a result, there is still a large performance gap between the
coalesced and shared memory bank conflict free transpose and the shared
Optimizing Matrix Transpose in CUDA

12
January 2009
memory copy.  In the next section we break the transpose into components to
determine the cause for the performance degradation.

Decomposing Transpose
There is over a factor of four performance difference between the best optimized
transpose and the shared memory copy in the table above.  This is the case not
only for measurements which loop over the kernel launches, but also for
measurements obtained from looping within the kernel where the costs associated
with the additional index calculations are amortized over the 100 iterations.
To investigate further, we revisit the data flow for the transpose and compare it to
that of the copy, both of which are indicated in the top portion of the diagram
below. There are essentially two differences between the copy code and the
transpose: transposing the data within a tile, and writing data to transposed tile.
We can isolate the performance between each of these two components by
implementing two kernels that individually perform just one of these
components.  As indicated in the bottom half of the diagram below, the fine-
grained transpose kernel transposes the data within a tile, but writes the tile to the
location that a copy would write the tile.  The coarse-grained transpose kernel
writes the tile to the transposed location in the odata matrix, but does not
transpose the data within the tile.

idata
odata
tile
copy
transpose
coarse-grained
transpose
fine-grained
transpose
Optimizing Matrix Transpose in CUDA

January 2009
  13

The source code for these two kernels is given below:
__global__ void transposeFineGrained(float *odata,
           float *idata, int width, int height,  int nreps)
{
  __shared__ float block[TILE_DIM][TILE_DIM+1];

  int xIndex = blockIdx.x * TILE_DIM + threadIdx.x;
  int yIndex = blockIdx.y * TILE_DIM + threadIdx.y;
  int index = xIndex + (yIndex)*width;

  for (int r=0; r<nreps; r++) {
    for (int i=0; i < TILE_DIM; i += BLOCK_ROWS) {
      block[threadIdx.y+i][threadIdx.x] =
        idata[index+i*width];
    }

    __syncthreads();

    for (int i=0; i < TILE_DIM; i += BLOCK_ROWS) {
      odata[index+i*height] =
        block[threadIdx.x][threadIdx.y+i];
    }
  }
}

__global__ void transposeCoarseGrained(float *odata,
      float *idata, int width, int height, int nreps)
{
  __shared__ float block[TILE_DIM][TILE_DIM+1];

  int xIndex = blockIdx.x * TILE_DIM + threadIdx.x;
  int yIndex = blockIdx.y * TILE_DIM + threadIdx.y;
  int index_in = xIndex + (yIndex)*width;

  xIndex = blockIdx.y * TILE_DIM + threadIdx.x;
  yIndex = blockIdx.x * TILE_DIM + threadIdx.y;
  int index_out = xIndex + (yIndex)*height;

  for (int r=0; r<nreps; r++) {
    for (int i=0; i<TILE_DIM; i += BLOCK_ROWS) {
      block[threadIdx.y+i][threadIdx.x] =
        idata[index_in+i*width];
    }

    __syncthreads();

    for (int i=0; i<TILE_DIM; i += BLOCK_ROWS) {
      odata[index_out+i*height] =
        block[threadIdx.y+i][threadIdx.x];
    }
  }
}

Optimizing Matrix Transpose in CUDA

14
January 2009
Note that the fine- and coarse-grained kernels are not actual transposes since in
either case odata is not a transpose of idata, but as you will see they are
useful in analyzing performance bottlenecks.  The performance results for these
two cases are added to our table below:

Effective Bandwidth (GB/s)
2048x2048, GTX 280

Loop over kernel
Loop in kernel
Simple Copy
96.9
81.6
Shared Memory Copy
80.9
81.1
Naïve Transpose
2.2
2.2
Coalesced Transpose
16.5
17.1
Bank Conflict Free Transpose
16.6
17.2
Fine-grained Transpose
80.4
81.5
Coarse-grained Transpose
16.7
17.1

The fine-grained transpose has performance similar to the shared memory copy,
whereas the coarse-grained transpose has roughly the performance of the
coalesced and bank conflict free transposes.  Thus the performance bottleneck
lies in writing data to the transposed location in global memory.  Just as shared
memory performance can be degraded via bank conflicts, an analogous
performance degradation can occur with global memory access through partition
camping, which we investigate next.
Partition Camping
Just as shared memory is divided into 16 banks of 32-bit width, global memory is
divided into either 6 partitions (on 8- and 9-series GPUs) or 8 partitions (on 200-
and 10-series GPUs) of 256-byte width.  We previously discussed that to use
shared memory effectively, threads within a half warp should access different
banks so that these accesses can occur simultaneously.  If threads within a half
warp access shared memory though only a few banks, then bank conflicts occur.
To use global memory effectively, concurrent accesses to global memory by all
active warps should be divided evenly amongst partitions.  The term partition
camping is used to describe the case when global memory accesses are directed
through a subset of partitions, causing requests to queue up at some partitions
while other partitions go unused.
Optimizing Matrix Transpose in CUDA

January 2009
  15
While coalescing concerns global memory accesses within a half warp, partition
camping concerns global memory accesses amongst active half warps.  Since
partition camping concerns how active thread blocks behave, the issue of how
thread blocks are scheduled on multiprocessors is important. When a kernel is
launched, the order in which blocks are assigned to multiprocessors is determined
by the one-dimensional block ID defined as:
bid = blockIdx.x + gridDim.x*blockIdx.y;
which is a row-major ordering of the blocks in the grid.  Once maximum
occupancy is reached, additional blocks are assigned to multiprocessors as
needed.  How quickly and the order in which blocks complete cannot be
determined, so active blocks are initially contiguous but become less contiguous
as execution of the kernel progresses.
If we return to our matrix transpose and look at how tiles in our 2048x2048
matrices map to partitions on a GTX 280, as depicted in the figure below, we
immediately see that partition camping is a problem.

With 8 partitions of 256-byte width, all data in strides of 2048 bytes (or 512
floats) map to the same partition.  Any float matrix with an integral multiple of
512 columns, such as our 2048x2048 matrix, will contain columns whose
elements map to a single partition.  With tiles of 32x32 floats (or 128x128 bytes),
whose one-dimensional block IDs are shown in the figure, all the data within the
first two columns of tiles map to the same partition, and likewise for other pairs
of tile columns (assuming the matrices are aligned to a partition segment).
Combining how the matrix elements map to partitions, and how blocks are
scheduled, we can see that concurrent blocks will be accessing tiles row-wise in
idata which will be roughly equally distributed amongst partitions, however
these blocks will access tiles column-wise in odata which will typically access
global memory through just a few partitions.
Having diagnosed the problem as partition camping, the question now turns to
what can be done about it.  Just as with shared memory, padding is an option.
Adding an additional 64 columns (one partition width) to odata will cause rows
of a tile to map sequentially to different partitions.  However, such padding can
become prohibitive to certain applications.  There is a simpler solution that
essentially involves rescheduling how blocks are executed.

…
130
129
128
69
68
67
66
65
64
5
4
3
2
1
0

69
5

68
4

…
67
3

130
66
2

129
65
1

128
64
0
idata
odata
Optimizing Matrix Transpose in CUDA

16
January 2009
Diagonal block reordering
While the programmer does not have direct control of the order in which blocks
are scheduled, which is determined by the value of the automatic kernel variable
blockIdx, the programmer does have the flexibility in how to interpret the
components of blockIdx.  Given how the components blockIdx are named,
i.e. x and y, one generally assumes these components refer to a cartesian
coordinate system.  This does not need to be the case, however, and one can
choose otherwise.  Within the cartesian interpretation one could swap the roles of
these two components, which would eliminate the partition camping problem in
writing to odata, however this would merely move the problem to reading data
from idata.
One way to avoid partition camping in both reading from idata and writing to
odata is to use a diagonal interpretation of the components of blockIdx: the
y component represents different diagonal slices of tiles through the matrix and
the x component indicates the distance along each diagonal.  Both cartesian and
diagonal interpretations of blockIdx components are shown in the top portion
of the diagram below for a 4x4-block matrix, along with the resulting one-
dimensional block ID on the bottom.

3,3
2,3
1,3
0,3
3,2
2,2
1,2
0,2
3,1
2,1
1,1
0,1
3,0
2,0
1,0
0,0
3,0
3,3
3,2
3,1
2,1
2,0
2,3
2,2
1,2
1,1
1,0
1,3
0,3
0,2
0,1
0,0
blockIdx.x + gridDim.x*blockIdx.y
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
0
3
15
11
7
6
2
14
10
9
5
1
13
12
8
4
0
Cartesian
Coordinate
Diagonal
Coordinate
Optimizing Matrix Transpose in CUDA

January 2009
  17
Before we discuss the merits of using the diagonal interpretation of blockIdx
components in the matrix transpose, we briefly mention how it can be efficiently
implemented using a mapping of coordinates.  This technique is useful when
writing new kernels, but even more so when modifying existing kernels to use
diagonal (or other) interpretations of blockIdx fields.  If blockIdx.x and
blockIdx.y represent the diagonal coordinates, then (for block-square
matrixes) the corresponding cartesian coordinates are given by the following
mapping:
blockIdx_y = blockIdx.x;
blockIdx_x = (blockIdx.x+blockIdx.y)%gridDim.x;
One would simply include the previous two lines of code at the beginning of the
kernel, and write the kernel assuming the cartesian interpretation of blockIdx
fields, except using blockIdx_x and blockIdx_y in place of blockIdx.x
and blockIdx.y, respectively, throughout the kernel.  This is precisely what is
done in the transposeDiagonal kernel below:

__global__ void transposeDiagonal(float *odata,
            float *idata, int width, int height, int nreps)
{
  __shared__ float tile[TILE_DIM][TILE_DIM+1];

  int blockIdx_x, blockIdx_y;

  // diagonal reordering
  if (width == height) {
    blockIdx_y = blockIdx.x;
    blockIdx_x = (blockIdx.x+blockIdx.y)%gridDim.x;
  } else {
    int bid = blockIdx.x + gridDim.x*blockIdx.y;
    blockIdx_y = bid%gridDim.y;
    blockIdx_x = ((bid/gridDim.y)+blockIdx_y)%gridDim.x;
  }

  int xIndex = blockIdx_x*TILE_DIM + threadIdx.x;
  int yIndex = blockIdx_y*TILE_DIM + threadIdx.y;
  int index_in = xIndex + (yIndex)*width;

  xIndex = blockIdx_y*TILE_DIM + threadIdx.x;
  yIndex = blockIdx_x*TILE_DIM + threadIdx.y;
  int index_out = xIndex + (yIndex)*height;

  for (int r=0; r < nreps; r++) {
    for (int i=0; i<TILE_DIM; i+=BLOCK_ROWS) {
      tile[threadIdx.y+i][threadIdx.x] =
        idata[index_in+i*width];
    }

    __syncthreads();

    for (int i=0; i<TILE_DIM; i+=BLOCK_ROWS) {
      odata[index_out+i*height] =
        tile[threadIdx.x][threadIdx.y+i];
    }
  }
}
Optimizing Matrix Transpose in CUDA

18
January 2009

Here we allow for both square and nonsquare matrices.  The mapping for
nonsquare matrices can be used in the general case, however the simpler
expressions for square matrices evaluate quicker and are preferable when
appropriate.
If we revisit our 2048x2048 matrix in the figure below, we can see how the
diagonal reordering solves the partition camping problem.  When reading from
idata and writing to odata in the diagonal case, pairs of tiles cycle through
partitions just as in the cartesian case when reading data from idata.

The performance of the diagonal kernel in the table below reflects this.  The
bandwidth measured when looping within the kernel over the read and writes to
global memory is within a few percent of the shared memory copy.  When
looping over the kernel, the performance degrades slightly, likely due to
additional computation involved in calculating blockIdx_x and
blockIdx_y.  However, even with this performance degradation the diagonal
transpose has over four times the bandwidth of the other complete transposes.

…
130
129
128
69
68
67
66
65
64
5
4
3
2
1
0

69
5

68
4

…
67
3

130
66
2

129
65
1

128
64
0
idata
odata
5

68
4

…
67
3

130
66
2

129
65
1

128
64
0
5
68
…

4
67
130

3
66
129

2
65
128

1
64

0
Cartesian
Diagonal
Optimizing Matrix Transpose in CUDA

January 2009
  19

Effective Bandwidth (GB/s)
2048x2048, GTX 280

Loop over kernel
Loop in kernel
Simple Copy
96.9
81.6
Shared Memory Copy
80.9
81.1
Naïve Transpose
2.2
2.2
Coalesced Transpose
16.5
17.1
Bank Conflict Free Transpose
16.6
17.2
Fine-grained Transpose
80.4
81.5
Coarse-grained Transpose
16.7
17.1
Diagonal
69.5
78.3

January 2009
  20
 Summary
In this paper we have discussed several aspects of GPU memory management
through a sequence of progressively optimized transpose kernels.  The sequence
is typical of performance tuning using CUDA.  The first step in improving
effective bandwidth is to ensure that global memory accesses are coalesced,
which can improve performance by an order of magnitude.
The second step was to look at shared memory bank conflicts.  In this study
eliminating shared memory bank conflicts appeared to have little effect on
performance, however that is largely due to when it was applied in relation to
other optimizations:  the effect of bank conflicts were masked by partition
camping.  By removing the padding of the shared memory array in the diagonally
reordered transpose, one can see that bank conflicts have a sizeable effect on
performance.
While coalescing and bank conflicts will remain relatively consistent as the
problem size varies, partition camping is dependent on problem size, and varies
across different generations of hardware.  The particular sized matrix in this
example will experience far less performance degradation due to partition
camping on a G80-based card due to the different number of partitions: 6
partitions on the 8-series rather than 8 on the 200-series.
The final version of the transpose kernel by no means represents the highest level
of optimization that can be achieved.  Tile size, number of elements per thread,
and instruction optimizations can improve performance, both of the transpose
and the copy kernels. But in the study we merely focused on the issues that have
the largest impact.

January 2009
  21
 Appendix A - Host Code
#include <stdio.h>

// kernels transpose/copy a tile of TILE_DIM x TILE_DIM elements
// using a TILE_DIM x BLOCK_ROWS thread block, so that each thread
// transposes TILE_DIM/BLOCK_ROWS elements.  TILE_DIM must be an
// integral multiple of BLOCK_ROWS

#define TILE_DIM 32
#define BLOCK_ROWS 8

// Number of repetitions used for timing.

#define NUM_REPS  100

int
main( int argc, char** argv)
{
  // set matrix size
  const int size_x = 2048, size_y = 2048;

  // kernel pointer and descriptor
  void (*kernel)(float *, float *, int, int, int);
  char *kernelName;

  // execution configuration parameters
  dim3 grid(size_x/TILE_DIM, size_y/TILE_DIM),
       threads(TILE_DIM,BLOCK_ROWS);

  // CUDA events
  cudaEvent_t start, stop;

  // size of memory required to store the matrix
  const int mem_size = sizeof(float) * size_x*size_y;

  // allocate host memory
  float *h_idata = (float*) malloc(mem_size);
  float *h_odata = (float*) malloc(mem_size);
  float *transposeGold = (float *) malloc(mem_size);
  float *gold;

  // allocate device memory
  float *d_idata, *d_odata;
  cudaMalloc( (void**) &d_idata, mem_size);
  cudaMalloc( (void**) &d_odata, mem_size);

  // initalize host data
  for(int i = 0; i < (size_x*size_y); ++i)
    h_idata[i] = (float) i;

  // copy host data to device
  cudaMemcpy(d_idata, h_idata, mem_size,
             cudaMemcpyHostToDevice );
Optimizing Matrix Transpose in CUDA

22
January 2009

  // Compute reference transpose solution
  computeTransposeGold(transposeGold, h_idata, size_x, size_y);

  // print out common data for all kernels
  printf("\nMatrix size: %dx%d, tile: %dx%d, block: %dx%d\n\n",

 size_x, size_y, TILE_DIM, TILE_DIM, TILE_DIM, BLOCK_ROWS);

  printf("Kernel\t\t\tLoop over kernel\tLoop within kernel\n");
  printf("------\t\t\t----------------\t------------------\n");

  //
  // loop over different kernels
  //

  for (int k = 0; k<8; k++) {
    // set kernel pointer
    switch (k) {
    case 0:
      kernel = ©
      kernelName = "simple copy           "; break;
    case 1:
      kernel = &copySharedMem;
      kernelName = "shared memory copy    "; break;
    case 2:
      kernel = &transposeNaive;
      kernelName = "naive transpose       "; break;
    case 3:
      kernel = &transposeCoalesced;
      kernelName = "coalesced transpose   "; break;
    case 4:
      kernel = &transposeNoBankConflicts;
      kernelName = "no bank conflict trans"; break;
    case 5:
      kernel = &transposeCoarseGrained;
      kernelName = "coarse-grained        "; break;
    case 6:
      kernel = &transposeFineGrained;
      kernelName = "fine-grained          "; break;
    case 7:
      kernel = &transposeDiagonal;
      kernelName = "diagonal transpose    "; break;
    }

    // set reference solution
    // NB: fine- and coarse-grained kernels are not full
    //     transposes, so bypass check
    if (kernel == &copy || kernel == &copySharedMem) {
      gold = h_idata;
    } else if (kernel == &transposeCoarseGrained ||
               kernel == &transposeFineGrained) {
      gold = h_odata;
    } else {
      gold = transposeGold;
    }

    // initialize events, EC parameters
    cudaEventCreate(&start);
    cudaEventCreate(&stop);

    // warmup to avoid timing startup
Optimizing Matrix Transpose in CUDA

January 2009
  23
    kernel<<<grid, threads>>>(d_odata, d_idata, size_x,size_y, 1);

    // take measurements for loop over kernel launches
    cudaEventRecord(start, 0);
    for (int i=0; i < NUM_REPS; i++) {
      kernel<<<grid, threads>>>(d_odata, d_idata,size_x,size_y,1);
    }
    cudaEventRecord(stop, 0);
    cudaEventSynchronize(stop);
    float outerTime;
    cudaEventElapsedTime(&outerTime, start, stop);

    cudaMemcpy(h_odata,d_odata, mem_size, cudaMemcpyDeviceToHost);
    int res = comparef(gold, h_odata, size_x*size_y);
    if (res != 1)
      printf("*** %s kernel FAILED ***\n", kernelName);

    // take measurements for loop inside kernel
    cudaEventRecord(start, 0);
    kernel<<<grid,threads>>>
        (d_odata, d_idata, size_x, size_y, NUM_REPS);
    cudaEventRecord(stop, 0);
    cudaEventSynchronize(stop);
    float innerTime;
    cudaEventElapsedTime(&innerTime, start, stop);

    cudaMemcpy(h_odata,d_odata, mem_size, cudaMemcpyDeviceToHost);
    res = comparef(gold, h_odata, size_x*size_y);
    if (res != 1)
      printf("*** %s kernel FAILED ***\n", kernelName);

    // report effective bandwidths
    float outerBandwidth =
       2.*1000*mem_size/(1024*1024*1024)/(outerTime/NUM_REPS);
    float innerBandwidth =
       2.*1000*mem_size/(1024*1024*1024)/(innerTime/NUM_REPS);
    printf("%s\t%5.2f GB/s\t\t%5.2f GB/s\n",
       kernelName, outerBandwidth, innerBandwidth);
  }

  // cleanup

  free(h_idata); free(h_odata); free(transposeGold);
  cudaFree(d_idata); cudaFree(d_odata);
  cudaEventDestroy(start); cudaEventDestroy(stop);

  return 0;
}

Optimizing Matrix Transpose in CUDA

24
January 2009

Notice
ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND
OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA
MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE
MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT,
MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE.
Information furnished is believed to be accurate and reliable. However, NVIDIA Corporation assumes no
responsibility for the consequences of use of such information or for any infringement of patents or other
rights of third parties that may result from its use. No license is granted by implication or otherwise under
any patent or patent rights of NVIDIA Corporation. Specifications mentioned in this publication are subject to
change without notice. This publication supersedes and replaces all information previously supplied. NVIDIA
Corporation products are not authorized for use as critical components in life support devices or systems
without express written approval of NVIDIA Corporation.
Trademarks
NVIDIA, the NVIDIA logo, and CUDA are trademarks or registered trademarks of NVIDIA Corporation in the
United States and other countries. Other company and product names may be trademarks of the respective
companies with which they are associated.
Macrovision Compliance Statement
NVIDIA Products that are Macrovision enabled can only be sold or distributed to buyers with a valid and
existing authorization from Macrovision to purchase and incorporate the device into buyer’s products.
Macrovision copy protection technology is protected by U.S. patent numbers 5,583,936; 6,516,132;
6,836,549; and 7,050,698 and other intellectual property rights. The use of Macrovision’s copy protection
technology in the device must be authorized by Macrovision and is intended for home and other limited pay-
per-view uses only, unless otherwise authorized in writing by Macrovision. Reverse engineering or
disassembly is prohibited.
Copyright
© 2009 NVIDIA Corporation. All rights reserved.