CUDA Shared Memory Swizzling

Introduction

When we write CUDA kernels that use shared memory, we have to be careful about shared memory bank conflicts. Having severe shared memory bank conflicts can introduce a significant performance penalty.

One simple way to deal with shared memory bank conflicts is to use padding. However, padding can waste shared memory and can have other drawbacks.

In this blog post, I would like to discuss how to deal with shared memory bank conflicts using swizzling. Swizzling is a more complicated technique that can be used to avoid shared memory bank conflicts without wasting shared memory.

CUDA Shared Memory Swizzling

Swizzling Example

When we use CUDA shared memory to cache data without using padding, it’s very common that either reading from or writing to shared memory by a warp can cause shared memory bank conflicts. Swizzling is a technique that rearranges the mapping of the shared memory index to avoid shared memory bank conflicts. Matrix transpose is a perfect example that can have shared memory bank conflicts if the implementation does not use padding or swizzling.

In the above example, the shared memory is a 2D array of float with size 32 × 16. In terms of matrix transpose, each warp reads a row of 32 values from the global memory and writes them to the shared memory with swizzling. There will be no shared memory bank conflicts when writing to the shared memory. To perform matrix transpose, each wrap reads two swizzled “columns” of 32 values from the shared memory and writes them to the global memory. For example, the swizzled column 0 and 1 are colored in yellow and cyan, respectively. In this way, there will be only one shared memory bank conflict when reading from the shared memory. Without using swizzling, there will be 16 (2-way) shared memory bank conflicts when reading from the shared memory. Of course, obviously, if the shared memory is a 2D array of float with size 32 × 32, there will be no shared memory bank conflicts when writing to the shared memory and reading from the shared memory.

Swizzling Formula

Given an array of T array[][NX] on shared memory, we define NX × sizeof(T) == SWIZZLE_SIZE. The allowed values of SWIZZLE_SIZE are powers of 2 that is larger than or equal to 32, such as 32, 64, 128, 256, …, etc.

Given the index [y][x] in T array[][NX], we can compute the swizzled index x_swz as follows:

Compute the index of the TC-byte chunk within SWIZZLE_SIZE-byte segment:i_chunk = (y × NX + x) × sizeof(T) / sizeof(TC)y_chunk = i / (SWIZZLE_SIZE / sizeof(TC))x_chunk = i % (SWIZZLE_SIZE / sizeof(TC))

Compute the swizzled index of TC-byte chunk using XOR operation:x_chunk_swz = y_chunk ^ x_chunk

Compute swizzled index:x_swz = x_chunk_swz × sizeof(TC) / sizeof(T) % NX + x % (sizeof(TC) / sizeof(T))

Swizzling Properties

This swizzling formula has the following properties:

The property 1 ensures that there will be no data loss during swizzling. The property 2 ensures that the index before and after swizzling will be one to one mapped.

Here I am going to show some informal mathematical proofs for the properties from the swizzling formula.

Proof

We will first prove the property 1.

$$\begin{align}x_{\text{chunk}}&= i_{\text{chunk}} \% (\text{SWIZZLE_SIZE} / \text{sizeof}(\text{TC})) \\&= \left(\left(y × \text{NX} + x\right) × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})\right) \% (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})) \\&= \left(y × \text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) + x × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})\right) \% (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})) \\&= \left(y × \text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) \% (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})) + x × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) \% (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})) \right) \% (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})) \\&= \left(x × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) \% (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})) \right) \% (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})) \\&= x × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) \% (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})) \\&= \left( x \% \text{NX} \right) × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) \\&= x × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) \\\end{align}$$

It seems that we have derived another equivalent formula for $x_{\text{chunk}}$. Note that $\text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})$ is a bit (right) shifting operation when $\text{sizeof}(\text{TC}) / \text{sizeof}(\text{T})$ is a power of 2.

$$\begin{align}y_{\text{chunk}}&= i_{\text{chunk}} / (\text{SWIZZLE_SIZE} / \text{sizeof}(\text{TC})) \\&= \left(\left(y × \text{NX} + x\right) × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})\right) / (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})) \\&= \left(y × \text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) + x × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})\right) / (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})) \\&= y × \text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) / (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}))  + x × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) / (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}))  \\&= y + x / \text{NX} \\&= y \\\end{align}$$

It seems that we have also derived another equivalent formula for $y_{\text{chunk}}$.

$$\begin{align}x_{\text{chunk_swz}}&= y_{\text{chunk}} \oplus x_{\text{chunk}} \\&= y \oplus \left( x × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) \right) \\&= y / (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})) × \text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) + \left( y \% (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})) \right) \oplus \left( x × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) \right) \\&= y / (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})) × \text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) + \left( \left( y \% \text{NX} \right) × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) \right) \oplus \left( x × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) \right) \\&= y / (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})) × \text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) + \left( y \% \text{NX} \right) \oplus x × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) \\\end{align}$$

Note that $\oplus$ is the bitwise XOR operation. If either $y_{\text{chunk}}$ or $x_{\text{chunk}}$ is a constant, the mapping is a one to one mapping.

$$\begin{align}x_{\text{swz}}&= x_{\text{chunk_swz}} × \text{sizeof}(\text{TC}) / \text{sizeof}(\text{T}) \% \text{NX} + x \% (\text{sizeof}(\text{TC}) / \text{sizeof}(\text{T})) \\\end{align}$$

Here the proof becomes a little bit informal.

Because a consecutive of $\text{sizeof}(\text{TC}) / \text{sizeof}(\text{T})$ $x$ values will be mapped to one unique trunk index $x_{\text{chunk}}$, the mapping between $x_{\text{chunk}}$ and $x_{\text{chunk_swz}}$ is a one to one mapping, one $x_{\text{chunk_swz}}$ value will map to one unique $x_{\text{chunk_swz}} × \text{sizeof}(\text{TC}) / \text{sizeof}(\text{T}) \% \text{NX}$ value. To create the one to one mapping between the swizzled index $x_{\text{swz}}$ and the original index $x$, the offset $x \% (\text{sizeof}(\text{TC}) / \text{sizeof}(\text{T}))$ is added. Therefore, the index before and after swizzling must be one to one mapped.

The property 2 is trivial to show.

The property 3 might be somewhat easier to see given the following expression for $x_{\text{swz}}$.

$$\begin{align}x_{\text{swz}}&= x_{\text{chunk_swz}} × \text{sizeof}(\text{TC}) / \text{sizeof}(\text{T}) \% \text{NX} + x \% (\text{sizeof}(\text{TC}) / \text{sizeof}(\text{T})) \\&= \left( y / (\text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC})) × \text{NX} × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) + \left( y \% \text{NX} \right) \oplus x × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) \right) × \text{sizeof}(\text{TC}) / \text{sizeof}(\text{T}) \% \text{NX} + x \% (\text{sizeof}(\text{TC}) / \text{sizeof}(\text{T})) \\&= \left( y \% \text{NX} \right) \oplus x × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) × \text{sizeof}(\text{TC}) / \text{sizeof}(\text{T}) \% \text{NX} + x \% (\text{sizeof}(\text{TC}) / \text{sizeof}(\text{T})) \\&= \left( y \% \text{NX} \right) \oplus x × \text{sizeof}(\text{T}) / \text{sizeof}(\text{TC}) × \text{sizeof}(\text{TC}) / \text{sizeof}(\text{T}) + x \% (\text{sizeof}(\text{TC}) / \text{sizeof}(\text{T})) \\\end{align}$$

Given any $x$ and any $\{y, y+1, y+2, \cdots, y+\text{NX}-1\}$, the number of unique swizzled index $x_{\text{swz}}$ is $\text{NX}$ which is maximized.

Examples

Matrix Transpose

In this example, we implemented matrix transpose CUDA kernels using shared memory in three different ways:

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367

#include <algorithm>#include <cassert>#include <chrono>#include <cstdio>#include <functional>#include <iomanip>#include <iostream>#include <random>#include <vector>#include <cuda_runtime.h>#define CHECK_CUDA_ERROR(val) check((val), #val, __FILE__, __LINE__)void check(cudaError_t err, char const* func, char const* file, int line){    if (err != cudaSuccess)    {        std::cerr << "CUDA Runtime Error at: " << file << ":" << line                  << std::endl;        std::cerr << cudaGetErrorString(err) << " " << func << std::endl;        std::exit(EXIT_FAILURE);    }}#define CHECK_LAST_CUDA_ERROR() check_last(__FILE__, __LINE__)void check_last(char const* file, int line){    cudaError_t const err{cudaGetLastError()};    if (err != cudaSuccess)    {        std::cerr << "CUDA Runtime Error at: " << file << ":" << line                  << std::endl;        std::cerr << cudaGetErrorString(err) << std::endl;        std::exit(EXIT_FAILURE);    }}template <class T>float measure_performance(std::function<T(cudaStream_t)> bound_function,                          cudaStream_t stream, size_t num_repeats = 10,                          size_t num_warmups = 10){    cudaEvent_t start, stop;    float time;    CHECK_CUDA_ERROR(cudaEventCreate(&start));    CHECK_CUDA_ERROR(cudaEventCreate(&stop));    for (size_t i{0}; i < num_warmups; ++i)    {        bound_function(stream);    }    CHECK_CUDA_ERROR(cudaStreamSynchronize(stream));    CHECK_CUDA_ERROR(cudaEventRecord(start, stream));    for (size_t i{0}; i < num_repeats; ++i)    {        bound_function(stream);    }    CHECK_CUDA_ERROR(cudaEventRecord(stop, stream));    CHECK_CUDA_ERROR(cudaEventSynchronize(stop));    CHECK_LAST_CUDA_ERROR();    CHECK_CUDA_ERROR(cudaEventElapsedTime(&time, start, stop));    CHECK_CUDA_ERROR(cudaEventDestroy(start));    CHECK_CUDA_ERROR(cudaEventDestroy(stop));    float const latency{time / num_repeats};    return latency;}constexpr size_t div_up(size_t a, size_t b) { return (a + b - 1) / b; }template <typename T, size_t BLOCK_TILE_SIZE_X = 32,          size_t BLOCK_TILE_SIZE_Y = 32, size_t BLOCK_TILE_SKEW_SIZE_X = 0>__global__ void transpose(T* output_matrix, T const* input_matrix, size_t M,                          size_t N){    // Waste some shared memory to avoid bank conflicts if    // BLOCK_TILE_SKEW_SIZE_X != 0.    __shared__ T        shm[BLOCK_TILE_SIZE_Y][BLOCK_TILE_SIZE_X + BLOCK_TILE_SKEW_SIZE_X];    // In some algorithms, such as matrix multiplication,    // a warp of threads have to access a column of the 2D matrix in the shared    // memory. Using the conventional index mapping, if the column size is not a    // multiple of the warp size, there will be bank conflicts.    size_t const input_matrix_from_idx_x{threadIdx.x + blockIdx.x * blockDim.x};    size_t const input_matrix_from_idx_y{threadIdx.y + blockIdx.y * blockDim.y};    size_t const input_matrix_from_idx{input_matrix_from_idx_x +                                       input_matrix_from_idx_y * N};    size_t const shm_to_idx_x{threadIdx.x};    size_t const shm_to_idx_y{threadIdx.y};    if ((input_matrix_from_idx_y < M) && (input_matrix_from_idx_x < N))    {        // Coalesced global memory access.        // No shared memory bank conflict.        shm[shm_to_idx_y][shm_to_idx_x] = input_matrix[input_matrix_from_idx];    }    // Make sure the buffer in a block is filled.    __syncthreads();    size_t const block_thread_idx{threadIdx.x + threadIdx.y * blockDim.x};    size_t const shm_from_idx_x{block_thread_idx / BLOCK_TILE_SIZE_Y};    size_t const shm_from_idx_y{block_thread_idx % BLOCK_TILE_SIZE_Y};    size_t const output_matrix_to_idx_x{shm_from_idx_y +                                        blockIdx.y * blockDim.y};    size_t const output_matrix_to_idx_y{shm_from_idx_x +                                        blockIdx.x * blockDim.x};    size_t const output_matrix_to_idx{output_matrix_to_idx_x +                                      output_matrix_to_idx_y * M};    if ((output_matrix_to_idx_y < N) && (output_matrix_to_idx_x < M))    {        // Coalesced global memory access.        // No shared memory bank conflict if BLOCK_TILE_SKEW_SIZE_X = 1.        output_matrix[output_matrix_to_idx] =            shm[shm_from_idx_y][shm_from_idx_x];    }}template <typename T, size_t BLOCK_TILE_SIZE_X = 32,          size_t BLOCK_TILE_SIZE_Y = 32>__global__ void transpose_swizzling(T* output_matrix, T const* input_matrix,                                    size_t M, size_t N){    __shared__ T shm[BLOCK_TILE_SIZE_Y][BLOCK_TILE_SIZE_X];    // In some algorithms, such as matrix multiplication,    // a warp of threads have to access a column of the 2D matrix in the shared    // memory. Using the conventional index mapping, if the column size is not a    // multiple of the warp size, there will be bank conflicts.    size_t const input_matrix_from_idx_x{threadIdx.x + blockIdx.x * blockDim.x};    size_t const input_matrix_from_idx_y{threadIdx.y + blockIdx.y * blockDim.y};    size_t const input_matrix_from_idx{input_matrix_from_idx_x +                                       input_matrix_from_idx_y * N};    size_t const shm_to_idx_x{threadIdx.x};    size_t const shm_to_idx_y{threadIdx.y};    size_t const shm_to_idx_x_swizzled{(shm_to_idx_x ^ shm_to_idx_y) %                                       BLOCK_TILE_SIZE_X};    if ((input_matrix_from_idx_y < M) && (input_matrix_from_idx_x < N))    {        // Coalesced global memory access.        // No shared memory bank conflict.        shm[shm_to_idx_y][shm_to_idx_x_swizzled] =            input_matrix[input_matrix_from_idx];    }    // Make sure the buffer in a block is filled.    __syncthreads();    size_t const block_thread_idx{threadIdx.x + threadIdx.y * blockDim.x};    size_t const shm_from_idx_x{block_thread_idx / BLOCK_TILE_SIZE_Y};    size_t const shm_from_idx_y{block_thread_idx % BLOCK_TILE_SIZE_Y};    size_t const shm_from_idx_x_swizzled{(shm_from_idx_x ^ shm_from_idx_y) %                                         BLOCK_TILE_SIZE_X};    size_t const output_matrix_to_idx_x{shm_from_idx_y +                                        blockIdx.y * blockDim.y};    size_t const output_matrix_to_idx_y{shm_from_idx_x +                                        blockIdx.x * blockDim.x};    size_t const output_matrix_to_idx{output_matrix_to_idx_x +                                      output_matrix_to_idx_y * M};    if ((output_matrix_to_idx_y < N) && (output_matrix_to_idx_x < M))    {        // Coalesced global memory access.        // No shared memory bank conflict.        output_matrix[output_matrix_to_idx] =            shm[shm_from_idx_y][shm_from_idx_x_swizzled];    }}template <typename T>void launch_transpose_with_shm_bank_conflict(T* d_output_matrix,                                             T const* d_input_matrix, size_t M,                                             size_t N, cudaStream_t stream){    constexpr size_t BLOCK_TILE_SIZE_X{32};    constexpr size_t BLOCK_TILE_SIZE_Y{32};    constexpr size_t BLOCK_TILE_SKEW_SIZE_X{0};    dim3 const block_size{BLOCK_TILE_SIZE_X, BLOCK_TILE_SIZE_Y};    dim3 const grid_size{static_cast<unsigned int>(div_up(N, block_size.x)),                         static_cast<unsigned int>(div_up(M, block_size.y))};    transpose<T, BLOCK_TILE_SIZE_X, BLOCK_TILE_SIZE_Y, BLOCK_TILE_SKEW_SIZE_X>        <<<grid_size, block_size, 0, stream>>>(d_output_matrix, d_input_matrix,                                               M, N);    CHECK_LAST_CUDA_ERROR();}template <typename T>void launch_transpose_without_shm_bank_conflict_via_padding(    T* d_output_matrix, T const* d_input_matrix, size_t M, size_t N,    cudaStream_t stream){    constexpr size_t BLOCK_TILE_SIZE_X{32};    constexpr size_t BLOCK_TILE_SIZE_Y{32};    constexpr size_t BLOCK_TILE_SKEW_SIZE_X{1};    dim3 const block_size{BLOCK_TILE_SIZE_X, BLOCK_TILE_SIZE_Y};    dim3 const grid_size{static_cast<unsigned int>(div_up(N, block_size.x)),                         static_cast<unsigned int>(div_up(M, block_size.y))};    transpose<T, BLOCK_TILE_SIZE_X, BLOCK_TILE_SIZE_Y, BLOCK_TILE_SKEW_SIZE_X>        <<<grid_size, block_size, 0, stream>>>(d_output_matrix, d_input_matrix,                                               M, N);    CHECK_LAST_CUDA_ERROR();}template <typename T>void launch_transpose_without_shm_bank_conflict_via_swizzling(    T* d_output_matrix, T const* d_input_matrix, size_t M, size_t N,    cudaStream_t stream){    constexpr size_t BLOCK_TILE_SIZE_X{32};    constexpr size_t BLOCK_TILE_SIZE_Y{32};    dim3 const block_size{BLOCK_TILE_SIZE_X, BLOCK_TILE_SIZE_Y};    dim3 const grid_size{static_cast<unsigned int>(div_up(N, block_size.x)),                         static_cast<unsigned int>(div_up(M, block_size.y))};    transpose_swizzling<T, BLOCK_TILE_SIZE_X, BLOCK_TILE_SIZE_Y><<<grid_size, block_size, 0, stream>>>(        d_output_matrix, d_input_matrix, M, N);    CHECK_LAST_CUDA_ERROR();}template <typename T>bool is_equal(T const* data_1, T const* data_2, size_t size){    for (size_t i{0}; i < size; ++i)    {        if (data_1[i] != data_2[i])        {            return false;        }    }    return true;}template <typename T>bool verify_transpose_implementation(    std::function<void(T*, T const*, size_t, size_t, cudaStream_t)>        transpose_function,    size_t M, size_t N){    // Fixed random seed for reproducibility    std::mt19937 gen{0};    cudaStream_t stream;    size_t const matrix_size{M * N};    std::vector<T> matrix(matrix_size, 0.0f);    std::vector<T> matrix_transposed(matrix_size, 1.0f);    std::vector<T> matrix_transposed_reference(matrix_size, 2.0f);    std::uniform_real_distribution<T> uniform_dist(-256, 256);    for (size_t i{0}; i < matrix_size; ++i)    {        matrix[i] = uniform_dist(gen);    }    // Create the reference transposed matrix using CPU.    for (size_t i{0}; i < M; ++i)    {        for (size_t j{0}; j < N; ++j)        {            size_t const from_idx{i * N + j};            size_t const to_idx{j * M + i};            matrix_transposed_reference[to_idx] = matrix[from_idx];        }    }    T* d_matrix;    T* d_matrix_transposed;    CHECK_CUDA_ERROR(cudaMalloc(&d_matrix, matrix_size * sizeof(T)));    CHECK_CUDA_ERROR(cudaMalloc(&d_matrix_transposed, matrix_size * sizeof(T)));    CHECK_CUDA_ERROR(cudaStreamCreate(&stream));    CHECK_CUDA_ERROR(cudaMemcpy(d_matrix, matrix.data(),                                matrix_size * sizeof(T),                                cudaMemcpyHostToDevice));    transpose_function(d_matrix_transposed, d_matrix, M, N, stream);    CHECK_CUDA_ERROR(cudaStreamSynchronize(stream));    CHECK_CUDA_ERROR(cudaMemcpy(matrix_transposed.data(), d_matrix_transposed,                                matrix_size * sizeof(T),                                cudaMemcpyDeviceToHost));    bool const correctness{is_equal(matrix_transposed.data(),                                    matrix_transposed_reference.data(),                                    matrix_size)};    CHECK_CUDA_ERROR(cudaFree(d_matrix));    CHECK_CUDA_ERROR(cudaFree(d_matrix_transposed));    CHECK_CUDA_ERROR(cudaStreamDestroy(stream));    return correctness;}template <typename T>float profile_transpose_implementation(    std::function<void(T*, T const*, size_t, size_t, cudaStream_t)>        transpose_function,    size_t M, size_t N){    constexpr int num_repeats{100};    constexpr int num_warmups{10};    cudaStream_t stream;    size_t const matrix_size{M * N};    T* d_matrix;    T* d_matrix_transposed;    CHECK_CUDA_ERROR(cudaMalloc(&d_matrix, matrix_size * sizeof(T)));    CHECK_CUDA_ERROR(cudaMalloc(&d_matrix_transposed, matrix_size * sizeof(T)));    CHECK_CUDA_ERROR(cudaStreamCreate(&stream));    std::function<void(cudaStream_t)> const transpose_function_wrapped{        std::bind(transpose_function, d_matrix_transposed, d_matrix, M, N,                  std::placeholders::_1)};    float const transpose_function_latency{measure_performance(        transpose_function_wrapped, stream, num_repeats, num_warmups)};    CHECK_CUDA_ERROR(cudaFree(d_matrix));    CHECK_CUDA_ERROR(cudaFree(d_matrix_transposed));    CHECK_CUDA_ERROR(cudaStreamDestroy(stream));    return transpose_function_latency;}void print_latencty(std::string const& kernel_name, float latency){    std::cout << kernel_name << ": " << std::fixed << std::setprecision(2)              << latency << " ms" << std::endl;}int main(){    // Unit tests.    for (size_t m{1}; m <= 64; ++m)    {        for (size_t n{1}; n <= 64; ++n)        {            assert(verify_transpose_implementation<float>(                &launch_transpose_with_shm_bank_conflict<float>, m, n));            assert(verify_transpose_implementation<float>(                &launch_transpose_without_shm_bank_conflict_via_padding<float>,                m, n));            assert(verify_transpose_implementation<float>(                &launch_transpose_without_shm_bank_conflict_via_swizzling<                    float>,                m, n));        }    }    // M: Number of rows.    size_t const M{8192};    // N: Number of columns.    size_t const N{8192};    std::cout << M << " x " << N << " Matrix" << std::endl;    float const latency_with_shm_bank_conflict{        profile_transpose_implementation<float>(            &launch_transpose_with_shm_bank_conflict<float>, M, N)};    print_latencty("Transpose with Shared Memory Bank Conflict",                   latency_with_shm_bank_conflict);    float const latency_without_shm_bank_conflict_via_padding{        profile_transpose_implementation<float>(            &launch_transpose_without_shm_bank_conflict_via_padding<float>, M,            N)};    print_latencty("Transpose without Shared Memory Bank Conflict via Padding",                   latency_without_shm_bank_conflict_via_padding);    float const latency_without_shm_bank_conflict_via_swizzling{        profile_transpose_implementation<float>(            &launch_transpose_without_shm_bank_conflict_via_swizzling<float>, M,            N)};    print_latencty(        "Transpose without Shared Memory Bank Conflict via Swizzling",        latency_without_shm_bank_conflict_via_swizzling);    return 0;}

The program was built and performed on a platform that has an Intel i9-9900K CPU and an NVIDIA RTX 3090 GPU.

123456

$ nvcc transpose.cu -o transpose$ ./transpose8192 x 8192 MatrixTranspose with Shared Memory Bank Conflict: 1.10 msTranspose without Shared Memory Bank Conflict via Padding: 0.92 msTranspose without Shared Memory Bank Conflict via Swizzling: 0.92 ms

We could see that the transpose kernel with shared memory bank conflict has the highest latency, while the transpose kernel without shared memory bank conflict via padding and swizzling have the same latency and run 20% faster than the kernel with shared memory bank conflict in this case.

Note that this implementation achieves ~65% of the peak memory bandwidth of an RTX 3090 GPU. The performance can be further improved significantly using vectorized memory access if the implementation assumes the matrix is always padded (and usually allocated using cudaMallocPitch) so that each row will continue to meet the coalescing requirement.

Swizzling vs Padding

Swizzling and padding are two common techniques to deal with shared memory bank conflicts.

The advantage of swizzling is that it does not waste shared memory space. The disadvantage of swizzling is that it is more complicated to implement and understand because the index mapping is not linear.

The advantage of padding is that it is simple to implement and understand. The disadvantage of padding is that it wastes shared memory space and can break the address alignment of the data if the padding size is not selected carefully especially when we access the data via large trunks using reinterpret_cast and cause undefined behaviors. This usually happens when vectorized memory access is performed on 2D padded arrays which accidentally breaks the alignment of the data.

References

CUDA Shared Memory Swizzling

https://leimao.github.io/blog/CUDA-Shared-Memory-Swizzling/

Author

Lei Mao

Posted on

05-14-2024

Updated on

07-31-2024

Licensed under

Like this article? Support the author with

Comments

Lei Mao

Artificial Intelligence
Machine Learning
Computer Science

Santa Clara, California

Posts

1008

Categories

7

Tags

662

Advertisement

Catalogue

© 2017-2025 Lei Mao  Powered by Hexo & IcarusSite UV:  Site PV:


CUDA Matrix Multiplication

Introduction

CUDA is a parallel computing platform and programming language that allows software to use certain types of graphics processing unit (GPU) for general purpose processing, an approach called general-purpose computing on GPUs (GPGPU). It could significantly enhance the performance of programs that could be computed with massive parallelism.

Matrix multiplication is a typical application that could be computed with massive parallelism. In this blog post, I would like to present a “hello-world” CUDA example of matrix multiplications and its preliminary optimizations.

Matrix Multiplication

There are two common matrix multiplication forms. The ordinary matrix multiplication mm and the batched matrix multiplication bmm.

$$\begin{align}\mathbf{C}^{n \times p} &= \mathbf{A}^{n \times m} \mathbf{B}^{m \times p} \\\mathbf{C}^{b \times n \times p} &= \mathbf{A}^{b \times n \times m} \mathbf{B}^{b \times m \times p} \\\end{align}$$

The reader could find the specifications of mm and bmm from PyTorch documentation torch.mm and torch.bmm.

In the following example, we first implemented the mm and bmm using C++. Then we implemented the mm using CUDA and naturally extended the mm implementation to the bmm implementation. Finally, we verified the correctness of the mm and bmm CUDA implementations.

Naive Implementation

This is the single source code file that contains the CPU and CUDA implementations for the matrix multiplication mm and the batched matrix multiplication bmm.

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477

#include <cassert>#include <cstddef>#include <cstdint>#include <iomanip>#include <iostream>#include <random>#include <stdexcept>#include <vector>#define BLOCK_DIM 32#define checkCuda(val) check((val), #val, __FILE__, __LINE__)void check(cudaError_t err, const char* const func, const char* const file,           const int line){    if (err != cudaSuccess)    {        std::cerr << "CUDA Runtime Error at: " << file << ":" << line                  << std::endl;        std::cerr << cudaGetErrorString(err) << " " << func << std::endl;        std::exit(EXIT_FAILURE);    }}template <typename T>std::vector<T> create_rand_vector(size_t n){    std::random_device r;    std::default_random_engine e(r());    std::uniform_int_distribution<int> uniform_dist(-256, 256);    std::vector<T> vec(n);    for (size_t i{0}; i < n; ++i)    {        vec.at(i) = static_cast<T>(uniform_dist(e));    }    return vec;}// mat_1: m x n// mat_2: n x p// mat_3: m x ptemplate <typename T>void mm(T const* mat_1, T const* mat_2, T* mat_3, size_t m, size_t n, size_t p){    // Compute the cells in mat_3 sequentially.    for (size_t i{0}; i < m; ++i)    {        for (size_t j{0}; j < p; ++j)        {            T acc_sum{0};            for (size_t k{0}; k < n; ++k)            {                acc_sum += mat_1[i * n + k] * mat_2[k * p + j];            }            mat_3[i * p + j] = acc_sum;        }    }}// mat_1: b x m x n// mat_2: b x n x p// mat_3: b x m x ptemplate <typename T>void bmm(T const* mat_1, T const* mat_2, T* mat_3, size_t b, size_t m, size_t n,         size_t p){    // Iterate through the batch dimension.    for (size_t i{0}; i < b; ++i)    {        mm(mat_1 + i * (m * n), mat_2 + i * (n * p), mat_3 + i * (m * p), m, n,           p);    }}template <typename T>__global__ void mm_kernel(T const* mat_1, T const* mat_2, T* mat_3, size_t m,                          size_t n, size_t p){    // 2D block and 2D thread    // Each thread computes one cell in mat_3.    size_t i{blockIdx.y * blockDim.y + threadIdx.y};    size_t j{blockIdx.x * blockDim.x + threadIdx.x};    // Do not process outside the matrix.    // Do not forget the equal sign!    if ((i >= m) || (j >= p))    {        return;    }    T acc_sum{0};    for (size_t k{0}; k < n; ++k)    {        acc_sum += mat_1[i * n + k] * mat_2[k * p + j];    }    mat_3[i * p + j] = acc_sum;}// It should be straightforward to extend a kernel to support batching.template <typename T>__global__ void bmm_kernel(T const* mat_1, T const* mat_2, T* mat_3, size_t b,                           size_t m, size_t n, size_t p){    // 2D block and 2D thread    // Each thread computes one cell in mat_3.    size_t i{blockIdx.y * blockDim.y + threadIdx.y};    size_t j{blockIdx.x * blockDim.x + threadIdx.x};    size_t l{blockIdx.z};    // Do not process outside the matrix.    // Do not forget the equal sign!    if ((i >= m) || (j >= p))    {        return;    }    T acc_sum{0};    for (size_t k{0}; k < n; ++k)    {        acc_sum += mat_1[l * m * n + i * n + k] * mat_2[l * n * p + k * p + j];    }    mat_3[l * m * p + i * p + j] = acc_sum;}template <typename T>void mm_cuda(T const* mat_1, T const* mat_2, T* mat_3, size_t m, size_t n,             size_t p){    dim3 threads_per_block(BLOCK_DIM, BLOCK_DIM);    dim3 blocks_per_grid(1, 1);    blocks_per_grid.x = std::ceil(static_cast<double>(p) /                                  static_cast<double>(threads_per_block.x));    blocks_per_grid.y = std::ceil(static_cast<double>(m) /                                  static_cast<double>(threads_per_block.y));    mm_kernel<<<blocks_per_grid, threads_per_block>>>(mat_1, mat_2, mat_3, m, n,                                                      p);}template <typename T>void bmm_cuda(T const* mat_1, T const* mat_2, T* mat_3, size_t b, size_t m,              size_t n, size_t p){    dim3 threads_per_block(BLOCK_DIM, BLOCK_DIM);    dim3 blocks_per_grid(1, 1, 1);    blocks_per_grid.x = std::ceil(static_cast<double>(p) /                                  static_cast<double>(threads_per_block.x));    blocks_per_grid.y = std::ceil(static_cast<double>(m) /                                  static_cast<double>(threads_per_block.y));    blocks_per_grid.z = b;    bmm_kernel<<<blocks_per_grid, threads_per_block>>>(mat_1, mat_2, mat_3, b,                                                       m, n, p);}template <typename T>bool allclose(std::vector<T> const& vec_1, std::vector<T> const& vec_2,              T const& abs_tol){    if (vec_1.size() != vec_2.size())    {        return false;    }    for (size_t i{0}; i < vec_1.size(); ++i)    {        if (std::abs(vec_1.at(i) - vec_2.at(i)) > abs_tol)        {            std::cout << vec_1.at(i) << " " << vec_2.at(i) << std::endl;            return false;        }    }    return true;}template <typename T>bool random_test_mm_cuda(size_t m, size_t n, size_t p){    std::vector<T> const mat_1_vec{create_rand_vector<T>(m * n)};    std::vector<T> const mat_2_vec{create_rand_vector<T>(n * p)};    std::vector<T> mat_3_vec(m * p);    std::vector<T> mat_4_vec(m * p);    T const* mat_1{mat_1_vec.data()};    T const* mat_2{mat_2_vec.data()};    T* mat_3{mat_3_vec.data()};    T* mat_4{mat_4_vec.data()};    mm(mat_1, mat_2, mat_3, m, n, p);    T *d_mat_1, *d_mat_2, *d_mat_4;    // Allocate device buffer.    checkCuda(cudaMalloc(&d_mat_1, sizeof(T) * mat_1_vec.size()));    checkCuda(cudaMalloc(&d_mat_2, sizeof(T) * mat_2_vec.size()));    checkCuda(cudaMalloc(&d_mat_4, sizeof(T) * mat_4_vec.size()));    // Copy data from host to device.    checkCuda(cudaMemcpy(d_mat_1, mat_1, sizeof(T) * mat_1_vec.size(),                         cudaMemcpyHostToDevice));    checkCuda(cudaMemcpy(d_mat_2, mat_2, sizeof(T) * mat_2_vec.size(),                         cudaMemcpyHostToDevice));    // Run matrix multiplication on GPU.    mm_cuda(d_mat_1, d_mat_2, d_mat_4, m, n, p);    cudaDeviceSynchronize();    cudaError_t err{cudaGetLastError()};    if (err != cudaSuccess)    {        std::cerr << "CUDA Matrix Multiplication kernel failed to execute."                  << std::endl;        std::cerr << cudaGetErrorString(err) << std::endl;        std::exit(EXIT_FAILURE);    }    // Copy data from device to host.    checkCuda(cudaMemcpy(mat_4, d_mat_4, sizeof(T) * mat_4_vec.size(),                         cudaMemcpyDeviceToHost));    // Free device buffer.    checkCuda(cudaFree(d_mat_1));    checkCuda(cudaFree(d_mat_2));    checkCuda(cudaFree(d_mat_4));    return allclose<T>(mat_3_vec, mat_4_vec, 1e-4);}template <typename T>bool random_test_bmm_cuda(size_t b, size_t m, size_t n, size_t p){    std::vector<T> const mat_1_vec{create_rand_vector<T>(b * m * n)};    std::vector<T> const mat_2_vec{create_rand_vector<T>(b * n * p)};    std::vector<T> mat_3_vec(b * m * p);    std::vector<T> mat_4_vec(b * m * p);    T const* mat_1{mat_1_vec.data()};    T const* mat_2{mat_2_vec.data()};    T* mat_3{mat_3_vec.data()};    T* mat_4{mat_4_vec.data()};    bmm(mat_1, mat_2, mat_3, b, m, n, p);    T *d_mat_1, *d_mat_2, *d_mat_4;    // Allocate device buffer.    checkCuda(cudaMalloc(&d_mat_1, sizeof(T) * mat_1_vec.size()));    checkCuda(cudaMalloc(&d_mat_2, sizeof(T) * mat_2_vec.size()));    checkCuda(cudaMalloc(&d_mat_4, sizeof(T) * mat_4_vec.size()));    // Copy data from host to device.    checkCuda(cudaMemcpy(d_mat_1, mat_1, sizeof(T) * mat_1_vec.size(),                         cudaMemcpyHostToDevice));    checkCuda(cudaMemcpy(d_mat_2, mat_2, sizeof(T) * mat_2_vec.size(),                         cudaMemcpyHostToDevice));    // Run matrix multiplication on GPU.    bmm_cuda(d_mat_1, d_mat_2, d_mat_4, b, m, n, p);    cudaDeviceSynchronize();    cudaError_t err{cudaGetLastError()};    if (err != cudaSuccess)    {        std::cerr << "CUDA Matrix Multiplication kernel failed to execute."                  << std::endl;        std::cerr << cudaGetErrorString(err) << std::endl;        std::exit(EXIT_FAILURE);    }    // Copy data from device to host.    checkCuda(cudaMemcpy(mat_4, d_mat_4, sizeof(T) * mat_4_vec.size(),                         cudaMemcpyDeviceToHost));    // Free device buffer.    checkCuda(cudaFree(d_mat_1));    checkCuda(cudaFree(d_mat_2));    checkCuda(cudaFree(d_mat_4));    return allclose<T>(mat_3_vec, mat_4_vec, 1e-4);}template <typename T>bool random_multiple_test_mm_cuda(size_t num_tests){    std::random_device r;    std::default_random_engine e(r());    std::uniform_int_distribution<int> uniform_dist(1, 256);    size_t m{0}, n{0}, p{0};    bool success{false};    for (size_t i{0}; i < num_tests; ++i)    {        m = static_cast<size_t>(uniform_dist(e));        n = static_cast<size_t>(uniform_dist(e));        p = static_cast<size_t>(uniform_dist(e));        success = random_test_mm_cuda<T>(m, n, p);        if (!success)        {            return false;        }    }    return true;}template <typename T>bool random_multiple_test_bmm_cuda(size_t num_tests){    std::random_device r;    std::default_random_engine e(r());    std::uniform_int_distribution<int> uniform_dist(1, 256);    size_t b{0}, m{0}, n{0}, p{0};    bool success{false};    for (size_t i{0}; i < num_tests; ++i)    {        b = static_cast<size_t>(uniform_dist(e));        m = static_cast<size_t>(uniform_dist(e));        n = static_cast<size_t>(uniform_dist(e));        p = static_cast<size_t>(uniform_dist(e));        success = random_test_bmm_cuda<T>(b, m, n, p);        if (!success)        {            return false;        }    }    return true;}template <typename T>float measure_latency_mm_cuda(size_t m, size_t n, size_t p, size_t num_tests,                              size_t num_warmups){    cudaEvent_t startEvent, stopEvent;    float time{0.0f};    checkCuda(cudaEventCreate(&startEvent));    checkCuda(cudaEventCreate(&stopEvent));    T *d_mat_1, *d_mat_2, *d_mat_4;    // Allocate device buffer.    checkCuda(cudaMalloc(&d_mat_1, sizeof(T) * m * n));    checkCuda(cudaMalloc(&d_mat_2, sizeof(T) * n * p));    checkCuda(cudaMalloc(&d_mat_4, sizeof(T) * m * p));    for (size_t i{0}; i < num_warmups; ++i)    {        mm_cuda(d_mat_1, d_mat_2, d_mat_4, m, n, p);    }    checkCuda(cudaEventRecord(startEvent, 0));    for (size_t i{0}; i < num_tests; ++i)    {        mm_cuda(d_mat_1, d_mat_2, d_mat_4, m, n, p);    }    checkCuda(cudaEventRecord(stopEvent, 0));    checkCuda(cudaEventSynchronize(stopEvent));    cudaError_t err{cudaGetLastError()};    if (err != cudaSuccess)    {        std::cerr << "CUDA Matrix Multiplication kernel failed to execute."                  << std::endl;        std::cerr << cudaGetErrorString(err) << std::endl;        std::exit(EXIT_FAILURE);    }    checkCuda(cudaEventElapsedTime(&time, startEvent, stopEvent));    // Free device buffer.    checkCuda(cudaFree(d_mat_1));    checkCuda(cudaFree(d_mat_2));    checkCuda(cudaFree(d_mat_4));    float latency{time / num_tests};    return latency;}template <typename T>float measure_latency_bmm_cuda(size_t b, size_t m, size_t n, size_t p,                               size_t num_tests, size_t num_warmups){    cudaEvent_t startEvent, stopEvent;    float time{0.0f};    checkCuda(cudaEventCreate(&startEvent));    checkCuda(cudaEventCreate(&stopEvent));    T *d_mat_1, *d_mat_2, *d_mat_4;    // Allocate device buffer.    checkCuda(cudaMalloc(&d_mat_1, sizeof(T) * b * m * n));    checkCuda(cudaMalloc(&d_mat_2, sizeof(T) * b * n * p));    checkCuda(cudaMalloc(&d_mat_4, sizeof(T) * b * m * p));    for (size_t i{0}; i < num_warmups; ++i)    {        bmm_cuda(d_mat_1, d_mat_2, d_mat_4, b, m, n, p);    }    checkCuda(cudaEventRecord(startEvent, 0));    for (size_t i{0}; i < num_tests; ++i)    {        bmm_cuda(d_mat_1, d_mat_2, d_mat_4, b, m, n, p);    }    checkCuda(cudaEventRecord(stopEvent, 0));    checkCuda(cudaEventSynchronize(stopEvent));    cudaError_t err{cudaGetLastError()};    if (err != cudaSuccess)    {        std::cerr << "CUDA Matrix Multiplication kernel failed to execute."                  << std::endl;        std::cerr << cudaGetErrorString(err) << std::endl;        std::exit(EXIT_FAILURE);    }    checkCuda(cudaEventElapsedTime(&time, startEvent, stopEvent));    // Free device buffer.    checkCuda(cudaFree(d_mat_1));    checkCuda(cudaFree(d_mat_2));    checkCuda(cudaFree(d_mat_4));    float latency{time / num_tests};    return latency;}int main(){    constexpr size_t num_tests{10};    assert(random_multiple_test_mm_cuda<int32_t>(num_tests));    assert(random_multiple_test_mm_cuda<float>(num_tests));    assert(random_multiple_test_mm_cuda<double>(num_tests));    assert(random_multiple_test_bmm_cuda<int32_t>(num_tests));    assert(random_multiple_test_bmm_cuda<float>(num_tests));    assert(random_multiple_test_bmm_cuda<double>(num_tests));    constexpr size_t num_measurement_tests{100};    constexpr size_t num_measurement_warmups{10};    size_t b{128}, m{1024}, n{1024}, p{1024};    float mm_cuda_int32_latency{measure_latency_mm_cuda<int32_t>(        m, n, p, num_measurement_tests, num_measurement_warmups)};    float mm_cuda_float_latency{measure_latency_mm_cuda<float>(        m, n, p, num_measurement_tests, num_measurement_warmups)};    float mm_cuda_double_latency{measure_latency_mm_cuda<double>(        m, n, p, num_measurement_tests, num_measurement_warmups)};    float bmm_cuda_int32_latency{measure_latency_bmm_cuda<int32_t>(        b, m, n, p, num_measurement_tests, num_measurement_warmups)};    float bmm_cuda_float_latency{measure_latency_bmm_cuda<float>(        b, m, n, p, num_measurement_tests, num_measurement_warmups)};    float bmm_cuda_double_latency{measure_latency_bmm_cuda<double>(        b, m, n, p, num_measurement_tests, num_measurement_warmups)};    std::cout << "Matrix Multiplication CUDA Latency" << std::endl;    std::cout << "m: " << m << " "              << "n: " << n << " "              << "p: " << p << std::endl;    std::cout << "INT32: " << std::fixed << std::setprecision(5)              << mm_cuda_int32_latency << " ms" << std::endl;    std::cout << "FLOAT: " << std::fixed << std::setprecision(5)              << mm_cuda_float_latency << " ms" << std::endl;    std::cout << "DOUBLE: " << std::fixed << std::setprecision(5)              << mm_cuda_double_latency << " ms" << std::endl;    std::cout << "Batched Matrix Multiplication CUDA Latency" << std::endl;    std::cout << "b: " << b << " "              << "m: " << m << " "              << "n: " << n << " "              << "p: " << p << std::endl;    std::cout << "INT32: " << std::fixed << std::setprecision(5)              << bmm_cuda_int32_latency << " ms" << std::endl;    std::cout << "FLOAT: " << std::fixed << std::setprecision(5)              << bmm_cuda_float_latency << " ms" << std::endl;    std::cout << "DOUBLE: " << std::fixed << std::setprecision(5)              << bmm_cuda_double_latency << " ms" << std::endl;}

Run Naive Example

Building and running the example requires an NVIDIA GPU. We also used NVIDIA official Docker container to set up the building environment.

To start the Docker container, please run the following command on the host computer.

1

$ docker run -it --rm --gpus all --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v $(pwd):/mnt -w /mnt nvcr.io/nvidia/cuda:11.7.1-devel-ubuntu22.04

To build and run the application, please run the following command in the Docker container.

12345678910111213

$ cd /mnt/$ nvcc mm.cu -o mm -std=c++14$ ./mmMatrix Multiplication CUDA Latencym: 1024 n: 1024 p: 1024INT32: 1.11436 msFLOAT: 0.98451 msDOUBLE: 4.10433 msBatched Matrix Multiplication CUDA Latencyb: 128 m: 1024 n: 1024 p: 1024INT32: 125.26781 msFLOAT: 124.67697 msDOUBLE: 487.87039 ms

We should expect no assertion error or any other kind of error for build and execution. The latencies were measured on a NVIDIA RTX 3090 GPU.

Matrix Multiplication Optimizations

The CUDA kernel optimization is usually all about how to accelerate the data traffic without affecting the number of math operations. To get the CUDA kernel fully optimized for GPU, the user would have to be very experienced with low-level GPU features and specifications and CUDA programming. But this does not prevent us from doing some preliminary optimization based on some shallow understandings of GPU.

Make Matrix Multiplication More Math-Bound

GPU is very friendly with math-bound operations. According to my previous blog post “Math-Bound VS Memory-Bound Operations”, if the number of operations remains the same and the number of memory IO bytes gets reduced, the operation will become more math-bound. That is to say, we want

$$\begin{gather}\frac{N_{\text{op}}}{N_{\text{byte}}} > \frac{\text{BW}_{\text{math}}}{\text{BW}_{\text{mem}}}\end{gather}$$

In our matrix multiplication naive CUDA implementation,

$$\begin{align}\mathbf{C}^{n \times p} &= \mathbf{A}^{n \times m} \mathbf{B}^{m \times p} \\\end{align}$$

We have to do $mnp$ multiplication and additions, $2mnp$ reads from memory, and $mp$ writes to memory. We could ignore the $mp$ writes from memory IO because the $2mnp$ reads is usually much more than the $mp$ writes.

Suppose we are doing FP32 matrix multiplication,

$$\begin{align}\frac{N_{\text{op}}}{N_{\text{byte}}}&= \frac{2 \times mnp}{2mnp \times 4} \\&= \frac{1}{4} \\\end{align}$$

For a modern GPU such as NVIDIA RTX 3090, for FP32 math,

$$\begin{align}\frac{\text{BW}_{\text{math}}}{\text{BW}_{\text{mem}}} &= \frac{35.58}{0.936} \\&= 38.0 \\\end{align}$$

We could see that the naive CUDA matrix multiplication implementation does not get even close to math-bound. Since $N_{\text{op}}$ should be a constant in matrix multiplication, let’s see if we could reduce $N_{\text{byte}}$ by caching.

Ideally, if we could cache the two full operand matrices $\mathbf{A}^{n \times m}$ and $\mathbf{B}^{m \times p}$, we could make the matrix multiplication most math-bound. However, since the caching size is limited and the implementation is supposed to support matrix multiplications with all different sizes, caching the full matrices is not technically possible.

Matrix Multiplication Decomposition

It is possible to decompose matrix multiplication mm into smaller matrix multiplications.

$$\mathbf{A} =\begin{bmatrix}\mathbf{A}_{1,1}^{d \times d} & \mathbf{A}_{1,2}^{d \times d} & \cdots & \mathbf{A}_{1,n/d}^{d \times d} \\\mathbf{A}_{2,1}^{d \times d} & \mathbf{A}_{2,2}^{d \times d} & \cdots & \mathbf{A}_{2,n/d}^{d \times d} \\\vdots & \vdots & \ddots & \vdots \\\mathbf{A}_{m/d,1}^{d \times d} & \mathbf{A}_{m/d,2}^{d \times d} & \cdots & \mathbf{A}_{m/d,n/d}^{d \times d} \\\end{bmatrix}$$

$$\mathbf{B} =\begin{bmatrix}\mathbf{B}_{1,1}^{d \times d} & \mathbf{B}_{1,2}^{d \times d} & \cdots & \mathbf{B}_{1,p/d}^{d \times d} \\\mathbf{B}_{2,1}^{d \times d} & \mathbf{B}_{2,2}^{d \times d} & \cdots & \mathbf{B}_{2,p/d}^{d \times d} \\\vdots & \vdots & \ddots & \vdots \\\mathbf{B}_{n/d,1}^{d \times d} & \mathbf{B}_{n/d,2}^{d \times d} & \cdots & \mathbf{B}_{n/d,p/d}^{d \times d} \\\end{bmatrix}$$

$$\mathbf{C} =\begin{bmatrix}\mathbf{C}_{1,1}^{d \times d} & \mathbf{C}_{1,2}^{d \times d} & \cdots & \mathbf{C}_{1,p/d}^{d \times d} \\\mathbf{C}_{2,1}^{d \times d} & \mathbf{C}_{2,2}^{d \times d} & \cdots & \mathbf{C}_{2,p/d}^{d \times d} \\\vdots & \vdots & \ddots & \vdots \\\mathbf{C}_{m/d,1}^{d \times d} & \mathbf{C}_{m/d,2}^{d \times d} & \cdots & \mathbf{C}_{m/d,p/d}^{d \times d} \\\end{bmatrix}$$

$$\mathbf{C}_{i,j}^{d \times d} = \sum_{k=1}^{n/d} \mathbf{A}_{i,k}^{d \times d} \mathbf{B}_{k,j}^{d \times d}$$

The decomposition does not alter the number of operations $N_{\text{op}}$.

$$\begin{align}N_{\text{op}} &= 2d^3 \left( \frac{n}{d} \right) \left( \frac{m}{d} \frac{p}{d}\right) \\&= 2mnp \\\end{align}$$

Because small matrices $\mathbf{A}_{i,k}^{d \times d}$ and $\mathbf{B}_{k,j}^{d \times d}$ could be cached, the memory IO bytes could be reduced, and the overall matrix multiplication could become more math bound. Let’s calculate how much memory IO bytes is needed in this case.

$$\begin{align}N_{\text{byte}} &= 2d^2 \times 4 \times  \left( \frac{n}{d} \right) \left( \frac{m}{d} \frac{p}{d}\right) \\&= \frac{8mnp}{d} \\\end{align}$$

Therefore,

$$\begin{align}\frac{N_{\text{op}}}{N_{\text{byte}}}&= \frac{2mnp}{\frac{8mnp}{d}} \\&= \frac{d}{4} \\\end{align}$$

Notice that when $d=1$, the matrix multiplication falls back to the naive matrix multiplication. When $d$ becomes larger, the implementation becomes more math-bound.

Optimized Implementation

The following implementation decomposed the matrix multiplication into multiple small matrix multiplications. The source code could be found on GitHub.

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356

#include <cassert>#include <cstddef>#include <cstdint>#include <iomanip>#include <iostream>#include <random>#include <stdexcept>#include <vector>#define BLOCK_DIM 32#define checkCuda(val) check((val), #val, __FILE__, __LINE__)void check(cudaError_t err, const char* const func, const char* const file,           const int line){    if (err != cudaSuccess)    {        std::cerr << "CUDA Runtime Error at: " << file << ":" << line                  << std::endl;        std::cerr << cudaGetErrorString(err) << " " << func << std::endl;        std::exit(EXIT_FAILURE);    }}template <typename T>std::vector<T> create_rand_vector(size_t n){    std::random_device r;    std::default_random_engine e(r());    std::uniform_int_distribution<int> uniform_dist(-256, 256);    std::vector<T> vec(n);    for (size_t i{0}; i < n; ++i)    {        vec.at(i) = static_cast<T>(uniform_dist(e));    }    return vec;}// mat_1: m x n// mat_2: n x p// mat_3: m x ptemplate <typename T>void mm(T const* mat_1, T const* mat_2, T* mat_3, size_t m, size_t n, size_t p){    // Compute the cells in mat_3 sequentially.    for (size_t i{0}; i < m; ++i)    {        for (size_t j{0}; j < p; ++j)        {            T acc_sum{0};            for (size_t k{0}; k < n; ++k)            {                acc_sum += mat_1[i * n + k] * mat_2[k * p + j];            }            mat_3[i * p + j] = acc_sum;        }    }}template <typename T>__global__ void mm_kernel(T const* mat_1, T const* mat_2, T* mat_3, size_t m,                          size_t n, size_t p){    // 2D block and 2D thread    // Each thread computes one cell in mat_3.    size_t i{blockIdx.y * blockDim.y + threadIdx.y};    size_t j{blockIdx.x * blockDim.x + threadIdx.x};    // Do not process outside the matrix.    // Do not forget the equal sign!    if ((i >= m) || (j >= p))    {        return;    }    T acc_sum{0};    for (size_t k{0}; k < n; ++k)    {        acc_sum += mat_1[i * n + k] * mat_2[k * p + j];    }    mat_3[i * p + j] = acc_sum;}template <typename T>__global__ void mm_kernel_optimized(T const* mat_1, T const* mat_2, T* mat_3,                                    size_t m, size_t n, size_t p){    __shared__ T mat_1_tile[BLOCK_DIM][BLOCK_DIM];    __shared__ T mat_2_tile[BLOCK_DIM][BLOCK_DIM];    T acc_sum{0};    for (size_t tile_idx{0};         tile_idx < ceilf(static_cast<float>(n) / BLOCK_DIM); ++tile_idx)    {        size_t i{blockIdx.y * blockDim.y + threadIdx.y};        size_t j{tile_idx * blockDim.x + threadIdx.x};        if ((i < m) && (j < n))        {            mat_1_tile[threadIdx.y][threadIdx.x] = mat_1[i * n + j];        }        else        {            mat_1_tile[threadIdx.y][threadIdx.x] = 0;        }        i = tile_idx * blockDim.y + threadIdx.y;        j = blockIdx.x * blockDim.x + threadIdx.x;        if ((i < n) && (j < p))        {            mat_2_tile[threadIdx.y][threadIdx.x] = mat_2[i * p + j];        }        else        {            mat_2_tile[threadIdx.y][threadIdx.x] = 0;        }        __syncthreads();        for (size_t k{0}; k < BLOCK_DIM; ++k)        {            acc_sum += mat_1_tile[threadIdx.y][k] * mat_2_tile[k][threadIdx.x];        }        __syncthreads();    }    // 2D block and 2D thread    // Each thread computes one cell in mat_3.    size_t i{blockIdx.y * blockDim.y + threadIdx.y};    size_t j{blockIdx.x * blockDim.x + threadIdx.x};    if ((i < m) && (j < p))    {        mat_3[i * p + j] = acc_sum;    }}template <typename T>void mm_cuda(T const* mat_1, T const* mat_2, T* mat_3, size_t m, size_t n,             size_t p,             void (*f)(T const*, T const*, T*, size_t, size_t, size_t)){    dim3 threads_per_block(BLOCK_DIM, BLOCK_DIM);    dim3 blocks_per_grid(1, 1);    blocks_per_grid.x = std::ceil(static_cast<double>(p) /                                  static_cast<double>(threads_per_block.x));    blocks_per_grid.y = std::ceil(static_cast<double>(m) /                                  static_cast<double>(threads_per_block.y));    f<<<blocks_per_grid, threads_per_block>>>(mat_1, mat_2, mat_3, m, n, p);}template <typename T>bool allclose(std::vector<T> const& vec_1, std::vector<T> const& vec_2,              T const& abs_tol){    if (vec_1.size() != vec_2.size())    {        return false;    }    for (size_t i{0}; i < vec_1.size(); ++i)    {        if (std::abs(vec_1.at(i) - vec_2.at(i)) > abs_tol)        {            std::cout << vec_1.at(i) << " " << vec_2.at(i) << std::endl;            return false;        }    }    return true;}template <typename T>bool random_test_mm_cuda(size_t m, size_t n, size_t p,                         void (*f)(T const*, T const*, T*, size_t, size_t,                                   size_t)){    std::vector<T> const mat_1_vec{create_rand_vector<T>(m * n)};    std::vector<T> const mat_2_vec{create_rand_vector<T>(n * p)};    std::vector<T> mat_3_vec(m * p);    std::vector<T> mat_4_vec(m * p);    T const* mat_1{mat_1_vec.data()};    T const* mat_2{mat_2_vec.data()};    T* mat_3{mat_3_vec.data()};    T* mat_4{mat_4_vec.data()};    mm(mat_1, mat_2, mat_3, m, n, p);    T *d_mat_1, *d_mat_2, *d_mat_4;    // Allocate device buffer.    checkCuda(cudaMalloc(&d_mat_1, sizeof(T) * mat_1_vec.size()));    checkCuda(cudaMalloc(&d_mat_2, sizeof(T) * mat_2_vec.size()));    checkCuda(cudaMalloc(&d_mat_4, sizeof(T) * mat_4_vec.size()));    // Copy data from host to device.    checkCuda(cudaMemcpy(d_mat_1, mat_1, sizeof(T) * mat_1_vec.size(),                         cudaMemcpyHostToDevice));    checkCuda(cudaMemcpy(d_mat_2, mat_2, sizeof(T) * mat_2_vec.size(),                         cudaMemcpyHostToDevice));    // Run matrix multiplication on GPU.    mm_cuda(d_mat_1, d_mat_2, d_mat_4, m, n, p, f);    cudaDeviceSynchronize();    cudaError_t err{cudaGetLastError()};    if (err != cudaSuccess)    {        std::cerr << "CUDA Matrix Multiplication kernel failed to execute."                  << std::endl;        std::cerr << cudaGetErrorString(err) << std::endl;        std::exit(EXIT_FAILURE);    }    // Copy data from device to host.    checkCuda(cudaMemcpy(mat_4, d_mat_4, sizeof(T) * mat_4_vec.size(),                         cudaMemcpyDeviceToHost));    // Free device buffer.    checkCuda(cudaFree(d_mat_1));    checkCuda(cudaFree(d_mat_2));    checkCuda(cudaFree(d_mat_4));    return allclose<T>(mat_3_vec, mat_4_vec, 1e-4);}template <typename T>bool random_multiple_test_mm_cuda(size_t num_tests,                                  void (*f)(T const*, T const*, T*, size_t,                                            size_t, size_t)){    std::random_device r;    std::default_random_engine e(r());    std::uniform_int_distribution<int> uniform_dist(1, 256);    size_t m{0}, n{0}, p{0};    bool success{false};    for (size_t i{0}; i < num_tests; ++i)    {        m = static_cast<size_t>(uniform_dist(e));        n = static_cast<size_t>(uniform_dist(e));        p = static_cast<size_t>(uniform_dist(e));        success = random_test_mm_cuda<T>(m, n, p, f);        if (!success)        {            return false;        }    }    return true;}template <typename T>float measure_latency_mm_cuda(size_t m, size_t n, size_t p, size_t num_tests,                              size_t num_warmups,                              void (*f)(T const*, T const*, T*, size_t, size_t,                                        size_t)){    cudaEvent_t startEvent, stopEvent;    float time{0.0f};    checkCuda(cudaEventCreate(&startEvent));    checkCuda(cudaEventCreate(&stopEvent));    T *d_mat_1, *d_mat_2, *d_mat_4;    // Allocate device buffer.    checkCuda(cudaMalloc(&d_mat_1, sizeof(T) * m * n));    checkCuda(cudaMalloc(&d_mat_2, sizeof(T) * n * p));    checkCuda(cudaMalloc(&d_mat_4, sizeof(T) * m * p));    for (size_t i{0}; i < num_warmups; ++i)    {        mm_cuda(d_mat_1, d_mat_2, d_mat_4, m, n, p, f);    }    checkCuda(cudaEventRecord(startEvent, 0));    for (size_t i{0}; i < num_tests; ++i)    {        mm_cuda(d_mat_1, d_mat_2, d_mat_4, m, n, p, f);    }    checkCuda(cudaEventRecord(stopEvent, 0));    checkCuda(cudaEventSynchronize(stopEvent));    cudaError_t err{cudaGetLastError()};    if (err != cudaSuccess)    {        std::cerr << "CUDA Matrix Multiplication kernel failed to execute."                  << std::endl;        std::cerr << cudaGetErrorString(err) << std::endl;        std::exit(EXIT_FAILURE);    }    checkCuda(cudaEventElapsedTime(&time, startEvent, stopEvent));    // Free device buffer.    checkCuda(cudaFree(d_mat_1));    checkCuda(cudaFree(d_mat_2));    checkCuda(cudaFree(d_mat_4));    float latency{time / num_tests};    return latency;}int main(){    constexpr size_t num_tests{10};    assert(random_multiple_test_mm_cuda<int32_t>(num_tests, mm_kernel));    assert(random_multiple_test_mm_cuda<float>(num_tests, mm_kernel));    assert(random_multiple_test_mm_cuda<double>(num_tests, mm_kernel));    assert(        random_multiple_test_mm_cuda<int32_t>(num_tests, mm_kernel_optimized));    assert(random_multiple_test_mm_cuda<float>(num_tests, mm_kernel_optimized));    assert(        random_multiple_test_mm_cuda<double>(num_tests, mm_kernel_optimized));    constexpr size_t num_measurement_tests{100};    constexpr size_t num_measurement_warmups{10};    const size_t m{1024}, n{1024}, p{1024};    float mm_cuda_int32_latency{measure_latency_mm_cuda<int32_t>(        m, n, p, num_measurement_tests, num_measurement_warmups, mm_kernel)};    float mm_cuda_float_latency{measure_latency_mm_cuda<float>(        m, n, p, num_measurement_tests, num_measurement_warmups, mm_kernel)};    float mm_cuda_double_latency{measure_latency_mm_cuda<double>(        m, n, p, num_measurement_tests, num_measurement_warmups, mm_kernel)};    std::cout << "Matrix Multiplication CUDA Latency" << std::endl;    std::cout << "m: " << m << " "              << "n: " << n << " "              << "p: " << p << std::endl;    std::cout << "INT32: " << std::fixed << std::setprecision(5)              << mm_cuda_int32_latency << " ms" << std::endl;    std::cout << "FLOAT: " << std::fixed << std::setprecision(5)              << mm_cuda_float_latency << " ms" << std::endl;    std::cout << "DOUBLE: " << std::fixed << std::setprecision(5)              << mm_cuda_double_latency << " ms" << std::endl;    mm_cuda_int32_latency = measure_latency_mm_cuda<int32_t>(        m, n, p, num_measurement_tests, num_measurement_warmups,        mm_kernel_optimized);    mm_cuda_float_latency = measure_latency_mm_cuda<float>(        m, n, p, num_measurement_tests, num_measurement_warmups,        mm_kernel_optimized);    mm_cuda_double_latency = measure_latency_mm_cuda<double>(        m, n, p, num_measurement_tests, num_measurement_warmups,        mm_kernel_optimized);    std::cout << "Optimized Matrix Multiplication CUDA Latency" << std::endl;    std::cout << "m: " << m << " "              << "n: " << n << " "              << "p: " << p << std::endl;    std::cout << "INT32: " << std::fixed << std::setprecision(5)              << mm_cuda_int32_latency << " ms" << std::endl;    std::cout << "FLOAT: " << std::fixed << std::setprecision(5)              << mm_cuda_float_latency << " ms" << std::endl;    std::cout << "DOUBLE: " << std::fixed << std::setprecision(5)              << mm_cuda_double_latency << " ms" << std::endl;}

Run Optimized Example

In the same Docker container, build and run the following application. We could see that the latency of INT32 and FP32 matrix multiplication got improved too different degrees.

123456789101112

$ nvcc mm_optimization.cu -o mm_optimization --std c++14$ ./mm_optimizationMatrix Multiplication CUDA Latencym: 1024 n: 1024 p: 1024INT32: 1.04373 msFLOAT: 1.02149 msDOUBLE: 3.83370 msOptimized Matrix Multiplication CUDA Latencym: 1024 n: 1024 p: 1024INT32: 0.84207 msFLOAT: 0.81759 msDOUBLE: 3.95231 ms

Miscellaneous

There are more subtle factors affecting the performance and there are more optimization opportunities to further optimize the matrix multiplication implementation. But those requires more thorough understanding of GPU and CUDA.

References

CUDA Matrix Multiplication

https://leimao.github.io/blog/CUDA-Matrix-Multiplication/

Author

Lei Mao

Posted on

03-21-2022

Updated on

03-04-2023

Licensed under

Like this article? Support the author with

Comments

Lei Mao

Artificial Intelligence
Machine Learning
Computer Science

Santa Clara, California

Posts

1008

Categories

7

Tags

662

Advertisement

Catalogue

© 2017-2025 Lei Mao  Powered by Hexo & IcarusSite UV:  Site PV:


Kernel Tuner

Guides

Features

Reference

Getting Started¶

So you have installed Kernel Tuner! That’s great! But now you’d like to get started tuning some GPU code.

Let’s say we have a simple CUDA kernel stored in a file called vector_add_kernel.cu:

__global__ void vector_add(float * c, float * a, float * b, int n) {
    int i = (blockIdx.x * blockDim.x) + threadIdx.x;

    if ( i < n ) {
        c[i] = a[i] + b[i];
    }
}

This kernel simply performs a point-wise addition of vectors a and b and stores the result in c.

To tune this kernel with Kernel Tuner, we are going to create the input and output data in Python using Numpy arrays.

import numpy as np
import kernel_tuner

size = 1000000
a = np.random.randn(size).astype(np.float32)
b = np.random.randn(size).astype(np.float32)
c = np.zeros_like(b)
n = np.int32(size)

To tell Kernel Tuner how it should call the kernel, we can create a list in Python that should correspond to
our CUDA kernel’s argument list with the same order and types.

args = [c, a, b, n]

So far, we have created the data structures needed by Kernel Tuner to call our kernel, but we have not yet specified what we
want Kernel Tuner to tune in our kernel. For that, we create a dictionary that we call tune_params, in which keys correspond
to tunable parameters in our kernel and the values are lists of values that these parameters may take.

tune_params = dict()
tune_params["block_size_x"] = [32, 64, 128, 256, 512, 1024]

In the code above, we have inserted a key into our dictionary, namely "block_size_x". This is a special name for a tunable
parameter that is recognized by Kernel Tuner to denote the size of our thread block in the x-dimension.
For a full list of special parameter names, please see the Parameter Vocabulary.

Alright, we are all set to start calling Kernel Tuner’s main function, which is called tune_kernel.

results, env = kernel_tuner.tune_kernel("vector_add", "vector_add_kernel.cu", size, args, tune_params)

In the above, tune_kernel takes five arguments:

The kernel name passed as a string

The filename of the kernel, also as a string

The problem_size, which corresponds to the total number of elements/threads in our kernel

The argument list used to call our kernel

The dictionary holding our tunable parameters

What happens how, is that Kernel Tuner will copy the our kernel’s input and output data to the GPU, iteratively compile and
benchmark our kernel for every possible combination of all values of all tunable parameters (a.k.a brute force tuning), and
return the benchmarking results as a list of dictionaries, along with an env dictionary that lists important information
about the hardware and software in which the benchmarking took place.

This wraps up the most basic use case of Kernel Tuner. There is a lot more functionality, which is explained in various
guides, examples, and feature articles.

© Copyright 2016-2024, Ben van Werkhoven, Alessio Sclocco, Stijn Heldens, Floris-Jan Willemsen, Willem-Jan Palenstijn, Bram Veenboer and Richard Schoonhoven.


CUDA SHARED MEMORY
NVIDIA Corporation
2
REVIEW (1 OF 2)
Difference between host and device
Host
CPU
Device
GPU
Using __global__
to declare a function as device code
Executes on the device
Called from the host (or possibly from other device code)
Passing parameters from host code to a device function
3
REVIEW (2 OF 2)
Basic device memory management
cudaMalloc()
cudaMemcpy()
cudaFree()
Launching parallel kernels
Launch N copies of add() with add<<<N,1>>>(…);
Use blockIdx.x to access block index
4
1D STENCIL
Consider applying a 1D stencil to a 1D array of elements
Each output element is the sum of input elements within a radius
If radius is 3, then each output element is the sum of 7 input elements:
radius
radius
5
IMPLEMENTING WITHIN A BLOCK
Each thread processes one output element
blockDim.x elements per block
Input elements are read several times
With radius 3, each input element is read seven times
6
SHARING DATA BETWEEN THREADS
Terminology: within a block, threads share data via shared memory
Extremely fast on-chip memory, user-managed
Declare using __shared__, allocated per block
Data is not visible to threads in other blocks
7
IMPLEMENTING WITH SHARED MEMORY
Cache data in shared memory
Read (blockDim.x + 2 * radius) input elements from global memory to shared memory
Compute blockDim.x output elements
Write blockDim.x output elements to global memory
Each block needs a halo of radius elements at each boundary
blockDim.x output elements
halo on left
halo on right
8
STENCIL KERNEL
__global__ void stencil_1d(int *in, int *out) {
__shared__ int temp[BLOCK_SIZE + 2 * RADIUS];
int gindex = threadIdx.x + blockIdx.x * blockDim.x;
int lindex = threadIdx.x + RADIUS;
// Read input elements into shared memory
temp[lindex] = in[gindex];
if (threadIdx.x < RADIUS) {
temp[lindex - RADIUS] = in[gindex - RADIUS];
temp[lindex + BLOCK_SIZE] =
in[gindex + BLOCK_SIZE];
}
9
STENCIL KERNEL
// Apply the stencil
int result = 0;
for (int offset = -RADIUS ; offset <= RADIUS ; offset++)
result += temp[lindex + offset];
// Store the result
out[gindex] = result;
}
10
DATA RACE!
The stencil example will not work…
Suppose thread 15 reads the halo before thread 0 has fetched
temp[lindex] = in[gindex];
if (threadIdx.x < RADIUS) {
temp[lindex – RADIUS] = in[gindex – RADIUS];
temp[lindex + BLOCK_SIZE] = in[gindex + BLOCK_SIZE];
}
int result = 0;
result += temp[lindex + 1];
Store at temp[18]
Load from temp[19]
Skipped, threadIdx > RADIUS
11
__SYNCTHREADS()
void
__syncthreads();
Synchronizes all threads within a block
Used to prevent RAW / WAR / WAW hazards
All threads must reach the barrier
In conditional code, the condition must be uniform across the block
12
STENCIL KERNEL
Stencil Kernel
__global__ void stencil_1d(int *in, int *out) {
__shared__ int temp[BLOCK_SIZE + 2 * RADIUS];
int gindex = threadIdx.x + blockIdx.x * blockDim.x;
int lindex = threadIdx.x + radius;
// Read input elements into shared memory
temp[lindex] = in[gindex];
if (threadIdx.x < RADIUS) {
temp[lindex – RADIUS] = in[gindex – RADIUS];
temp[lindex + BLOCK_SIZE] = in[gindex + BLOCK_SIZE];
}
// Synchronize (ensure all the data is available)
__syncthreads();
13
STENCIL KERNEL
Stencil Kernel
// Apply the stencil
int result = 0;
for (int offset = -RADIUS ; offset <= RADIUS ; offset++)
result += temp[lindex + offset];
// Store the result
out[gindex] = result;
}
14
REVIEW
Use __shared__ to declare a variable/array in shared memory
Data is shared between threads in a block
Not visible to threads in other blocks
Use __syncthreads() as a barrier
Use to prevent data hazards
15
DEVELOPERS
Scalable Cooperation among groups of threads
Flexible parallel decompositions
Composition across software boundaries
Deploy Everywhere
Examples include:
Persistent RNNs
Physics
Search Algorithms
Sorting
Cooperative Groups: a flexible model for synchronization and
communication within groups of threads.
At a glance
Benefits all applications
LOOKING FORWARD
16
FOR EXAMPLE: THREAD BLOCK
Implicit group of all the threads in the launched thread block
Implements the same interface as thread_group:
void sync();
// Synchronize the threads in the group
unsigned size();
// Total number of threads in the group
unsigned thread_rank();
// Rank of the calling thread within [0, size)
bool is_valid();
// Whether the group violated any API constraints
And additional thread_block specific functions:
dim3 group_index();
// 3-dimensional block index within the grid
dim3 thread_index();
// 3-dimensional thread index within the block
17
NARROWING THE SHARED MEMORY GAP
with the GV100 L1 cache
Pascal
Volta
Cache: vs shared
•
Easier to use
•
90%+ as good
Shared: vs cache
•
Faster atomics
•
More banks
•
More predictable
Average
Shared
Memory
Benefit
70%
93%
Directed testing: shared in global
18
FUTURE SESSIONS
CUDA GPU architecture and basic optimizations
Atomics, Reductions, Warp Shuffle
Using Managed Memory
Concurrency (streams, copy/compute overlap, multi-GPU)
Analysis Driven Optimization
Cooperative Groups
19
FURTHER STUDY
Shared memory:
https://devblogs.nvidia.com/using-shared-memory-cuda-cc/
CUDA Programming Guide:
https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#shared-memory
CUDA Documentation:
https://docs.nvidia.com/cuda/index.html
https://docs.nvidia.com/cuda/cuda-runtime-api/index.html (runtime API)
20
HOMEWORK
Log into Summit (ssh username@home.ccs.ornl.gov -> ssh summit)
Clone GitHub repository:
Git clone git@github.com:olcf/cuda-training-series.git
Follow the instructions in the readme.md file:
https://github.com/olcf/cuda-training-series/blob/master/exercises/hw2/readme.md
Prerequisites: basic linux skills, e.g. ls, cd, etc., knowledge of a text editor like vi/emacs, and some
knowledge of C/C++ programming
QUESTIONS?


CUDA Execution Provider

The CUDA Execution Provider enables hardware accelerated computation on Nvidia CUDA-enabled GPUs.

Contents

Install

Pre-built binaries of ONNX Runtime with CUDA EP are published for most language bindings. Please reference Install ORT.

Build from source

See Build instructions.

Requirements

Please reference table below for official GPU packages dependencies for the ONNX Runtime inferencing package. Note that ONNX Runtime Training is aligned with PyTorch CUDA versions; refer to the Optimize Training tab on onnxruntime.ai for supported versions.

Because of Nvidia CUDA Minor Version Compatibility, ONNX Runtime built with CUDA 11.8 are compatible with any CUDA 11.x version; ONNX Runtime built with CUDA 12.x are compatible with any CUDA 12.x version.

ONNX Runtime built with cuDNN 8.x is not compatible with cuDNN 9.x, and vice versa. You can choose the package based on CUDA and cuDNN major versions that match your runtime environment (e.g., PyTorch 2.3 uses cuDNN 8.x, while PyTorch 2.4 or later uses cuDNN 9.x).

Note: Starting with version 1.19, CUDA 12.x becomes the default version when distributing ONNX Runtime GPU packages in PyPI.

CUDA 12.x

CUDA 11.x

CUDA 10.x

For older versions, please reference the readme and build pages on the release branch.

Build

For build instructions, please see the BUILD page.

Configuration Options

The CUDA Execution Provider supports the following configuration options.

device_id

The device ID.

Default value: 0

user_compute_stream

Defines the compute stream for the inference to run on. It implicitly sets the has_user_compute_stream option. It cannot be set through UpdateCUDAProviderOptions, but rather UpdateCUDAProviderOptionsWithValue. This cannot be used in combination with an external allocator.

Example python usage:

providers = [("CUDAExecutionProvider", {"device_id": torch.cuda.current_device(),
                                        "user_compute_stream": str(torch.cuda.current_stream().cuda_stream)})]
sess_options = ort.SessionOptions()
sess = ort.InferenceSession("my_model.onnx", sess_options=sess_options, providers=providers)

To take advantage of user compute stream, it is recommended to use I/O Binding to bind inputs and outputs to tensors in device.

do_copy_in_default_stream

Whether to do copies in the default stream or use separate streams. The recommended setting is true. If false, there are race conditions and possibly better performance.

Default value: true

use_ep_level_unified_stream

Uses the same CUDA stream for all threads of the CUDA EP. This is implicitly enabled by has_user_compute_stream, enable_cuda_graph or when using an external allocator.

Default value: false

gpu_mem_limit

The size limit of the device memory arena in bytes. This size limit is only for the execution provider’s arena. The total device memory usage may be higher. s: max value of C++ size_t type (effectively unlimited)

Note: Will be over-ridden by contents of default_memory_arena_cfg (if specified)

arena_extend_strategy

The strategy for extending the device memory arena.

Default value: kNextPowerOfTwo

Note: Will be over-ridden by contents of default_memory_arena_cfg (if specified)

cudnn_conv_algo_search

The type of search done for cuDNN convolution algorithms.

Default value: EXHAUSTIVE

cudnn_conv_use_max_workspace

Check tuning performance for convolution heavy models for details on what this flag does. This flag is only supported from the V2 version of the provider options struct when used using the C API.(sample below)

Default value: 1, for versions 1.14 and later 0, for previous versions

cudnn_conv1d_pad_to_nc1d

Check convolution input padding in the CUDA EP for details on what this flag does. This flag is only supported from the V2 version of the provider options struct when used using the C API. (sample below)

Default value: 0

enable_cuda_graph

Check using CUDA Graphs in the CUDA EP for details on what this flag does. This flag is only supported from the V2 version of the provider options struct when used using the C API. (sample below)

Default value: 0

enable_skip_layer_norm_strict_mode

Whether to use strict mode in SkipLayerNormalization cuda implementation. The default and recommended setting is false. If enabled, accuracy improvement and performance drop can be expected. This flag is only supported from the V2 version of the provider options struct when used using the C API. (sample below)

Default value: 0

use_tf32

TF32 is a math mode available on NVIDIA GPUs since Ampere. It allows certain float32 matrix multiplications and convolutions to run much faster on tensor cores with TensorFloat-32 reduced precision: float32 inputs are rounded with 10 bits of mantissa and results are accumulated with float32 precision.

Default value: 1

TensorFloat-32 is enabled by default. Starting from ONNX Runtime 1.18, you can use this flag to disable it for an inference session.

Example python usage:

providers = [("CUDAExecutionProvider", {"use_tf32": 0})]
sess_options = ort.SessionOptions()
sess = ort.InferenceSession("my_model.onnx", sess_options=sess_options, providers=providers)

This flag is only supported from the V2 version of the provider options struct when used using the C API. (sample below)

gpu_external_[alloc|free|empty_cache]

gpu_external_* is used to pass external allocators. Example python usage:

from onnxruntime.training.ortmodule.torch_cpp_extensions import torch_gpu_allocator

provider_option_map["gpu_external_alloc"] = str(torch_gpu_allocator.gpu_caching_allocator_raw_alloc_address())
provider_option_map["gpu_external_free"] = str(torch_gpu_allocator.gpu_caching_allocator_raw_delete_address())
provider_option_map["gpu_external_empty_cache"] = str(torch_gpu_allocator.gpu_caching_allocator_empty_cache_address())

Default value: 0

prefer_nhwc

This option is not available in default builds ! One has to compile ONNX Runtime with onnxruntime_USE_CUDA_NHWC_OPS=ON. If this is enabled the EP prefers NHWC operators over NCHW. Needed transforms will be added to the model. As NVIDIA tensor cores can only work on NHWC layout this can increase performance if the model consists of many supported operators and does not need too many new transpose nodes. Wider operator support is planned in the future.

This flag is only supported from the V2 version of the provider options struct when used using the C API. The V2 provider options struct can be created using CreateCUDAProviderOptions and updated using UpdateCUDAProviderOptions.

Default value: 0

Performance Tuning

The I/O Binding feature should be utilized to avoid overhead resulting from copies on inputs and outputs. Ideally up and downloads for inputs can be hidden behind the inference. This can be achieved by doing asynchronous copies while running inference. This is demonstrated in this PR

Ort::RunOptions run_options;
run_options.AddConfigEntry("disable_synchronize_execution_providers", "1");
session->Run(run_options, io_binding);

By disabling the synchronization on the inference the user has to take care of synchronizing the compute stream after execution. This feature should only be used with device local memory or an ORT Value allocated in pinned memory, otherwise the issued download will be blocking and not behave as desired.

Convolution-heavy models

ORT leverages CuDNN for convolution operations and the first step in this process is to determine which “optimal” convolution algorithm to use while performing the convolution operation for the given input configuration (input shape, filter shape, etc.) in each Conv node. This sub-step involves querying CuDNN for a “workspace” memory size and have this allocated so that CuDNN can use this auxiliary memory while determining the “optimal” convolution algorithm to use.

The default value of cudnn_conv_use_max_workspace is 1 for versions 1.14 or later, and 0 for previous versions. When its value is 0, ORT clamps the workspace size to 32 MB which may lead to a sub-optimal convolution algorithm getting picked by CuDNN. To allow ORT to allocate the maximum possible workspace as determined by CuDNN, a provider option named cudnn_conv_use_max_workspace needs to get set (as shown below).

Keep in mind that using this flag may increase the peak memory usage by a factor (sometimes a few GBs) but this does help CuDNN pick the best convolution algorithm for the given input. We have found that this is an important flag to use while using an fp16 model as this allows CuDNN to pick tensor core algorithms for the convolution operations (if the hardware supports tensor core operations). This flag may or may not result in performance gains for other data types (float and double).

providers = [("CUDAExecutionProvider", {"cudnn_conv_use_max_workspace": '1'})]
  sess_options = ort.SessionOptions()
  sess = ort.InferenceSession("my_conv_heavy_fp16_model.onnx", sess_options=sess_options, providers=providers)

OrtCUDAProviderOptionsV2* cuda_options = nullptr;
  CreateCUDAProviderOptions(&cuda_options);

  std::vector<const char*> keys{"cudnn_conv_use_max_workspace"};
  std::vector<const char*> values{"1"};

  UpdateCUDAProviderOptions(cuda_options, keys.data(), values.data(), 1);

  OrtSessionOptions* session_options = /* ... */;
  SessionOptionsAppendExecutionProvider_CUDA_V2(session_options, cuda_options);

  // Finally, don't forget to release the provider options
  ReleaseCUDAProviderOptions(cuda_options);

var cudaProviderOptions = new OrtCUDAProviderOptions(); // Dispose this finally

 var providerOptionsDict = new Dictionary<string, string>();
 providerOptionsDict["cudnn_conv_use_max_workspace"] = "1";

 cudaProviderOptions.UpdateOptions(providerOptionsDict);

 SessionOptions options = SessionOptions.MakeSessionOptionWithCudaProvider(cudaProviderOptions);  // Dispose this finally

Convolution Input Padding

ORT leverages CuDNN for convolution operations. While CuDNN only takes 4-D or 5-D tensor as input for convolution operations, dimension padding is needed if the input is 3-D tensor. Given an input tensor of shape [N, C, D], it can be padded to [N, C, D, 1] or [N, C, 1, D]. While both of these two padding ways produce same output, the performance may be a lot different because different convolution algorithms are selected, especially on some devices such as A100. By default the input is padded to [N, C, D, 1]. A provider option named cudnn_conv1d_pad_to_nc1d needs to get set (as shown below) if [N, C, 1, D] is preferred.

providers = [("CUDAExecutionProvider", {"cudnn_conv1d_pad_to_nc1d": '1'})]
  sess_options = ort.SessionOptions()
  sess = ort.InferenceSession("my_conv_model.onnx", sess_options=sess_options, providers=providers)

OrtCUDAProviderOptionsV2* cuda_options = nullptr;
  CreateCUDAProviderOptions(&cuda_options);

  std::vector<const char*> keys{"cudnn_conv1d_pad_to_nc1d"};
  std::vector<const char*> values{"1"};

  UpdateCUDAProviderOptions(cuda_options, keys.data(), values.data(), 1);

  OrtSessionOptions* session_options = /* ... */;
  SessionOptionsAppendExecutionProvider_CUDA_V2(session_options, cuda_options);

  // Finally, don't forget to release the provider options
  ReleaseCUDAProviderOptions(cuda_options);

var cudaProviderOptions = new OrtCUDAProviderOptions(); // Dispose this finally

  var providerOptionsDict = new Dictionary<string, string>();
  providerOptionsDict["cudnn_conv1d_pad_to_nc1d"] = "1";

  cudaProviderOptions.UpdateOptions(providerOptionsDict);

  SessionOptions options = SessionOptions.MakeSessionOptionWithCudaProvider(cudaProviderOptions);  // Dispose this finally

Using CUDA Graphs (Preview)

While using the CUDA EP, ORT supports the usage of CUDA Graphs to remove CPU overhead associated with launching CUDA kernels sequentially. To enable the usage of CUDA Graphs, use the provider options as shown in the samples below. ORT supports multi-graph capture capability by passing the user specified gpu_graph_id to the run options. gpu_graph_id is optional when the session uses one cuda graph. If not set, the default value is 0. If the gpu_graph_id is set to -1, cuda graph capture/replay is disabled in that run.

Currently, there are some constraints with regards to using the CUDA Graphs feature:

Models with control-flow ops (i.e. If, Loop and Scan ops) are not supported.

Usage of CUDA Graphs is limited to models where-in all the model ops (graph nodes) can be partitioned to the CUDA EP.

The input/output types of models need to be tensors.

Shapes and addresses of inputs/outputs cannot change across inference calls for the same graph annotation id. Input tensors for replay shall be copied to the address of input tensors used in graph capture.

In multi-graph capture mode, the captured graphs will remain in the session’s lifetime and the captured graph deletion feature is not supported at the moment.

By design, CUDA Graphs is designed to read from/write to the same CUDA virtual memory addresses during the graph replaying step as it does during the graph capturing step. Due to this requirement, usage of this feature requires using IOBinding so as to bind memory which will be used as input(s)/output(s) for the CUDA Graph machinery to read from/write to (please see samples below).

While updating the input(s) for subsequent inference calls, the fresh input(s) need to be copied over to the corresponding CUDA memory location(s) of the bound OrtValue input(s) (please see samples below to see how this can be achieved). This is due to the fact that the “graph replay” will require reading inputs from the same CUDA virtual memory addresses.

Multi-threaded usage is currently not supported, i.e. Run() MAY NOT be invoked on the same InferenceSession object from multiple threads while using CUDA Graphs.

NOTE: The very first Run() performs a variety of tasks under the hood like making CUDA memory allocations, capturing the CUDA graph for the model, and then performing a graph replay to ensure that the graph runs. Due to this, the latency associated with the first Run() is bound to be high. Subsequent Run()s only perform graph replays of the graph captured and cached in the first Run().

Python

providers = [("CUDAExecutionProvider", {"enable_cuda_graph": '1'})]
  sess_options = ort.SessionOptions()
  sess = ort.InferenceSession("my_model.onnx", sess_options=sess_options, providers=providers)

  providers = [("CUDAExecutionProvider", {'enable_cuda_graph': True})]
  x = np.array([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]], dtype=np.float32)
  y = np.array([[0.0], [0.0], [0.0]], dtype=np.float32)
  x_ortvalue = onnxrt.OrtValue.ortvalue_from_numpy(x, 'cuda', 0)
  y_ortvalue = onnxrt.OrtValue.ortvalue_from_numpy(y, 'cuda', 0)

  session = onnxrt.InferenceSession("matmul_2.onnx", providers=providers)
  io_binding = session.io_binding()

  # Pass gpu_graph_id to RunOptions through RunConfigs
  ro = onnxrt.RunOptions()
  # gpu_graph_id is optional if the session uses only one cuda graph
  ro.add_run_config_entry("gpu_graph_id", "1")

  # Bind the input and output
  io_binding.bind_ortvalue_input('X', x_ortvalue)
  io_binding.bind_ortvalue_output('Y', y_ortvalue)

  # One regular run for the necessary memory allocation and cuda graph capturing
  session.run_with_iobinding(io_binding, ro)
  expected_y = np.array([[5.0], [11.0], [17.0]], dtype=np.float32)
  np.testing.assert_allclose(expected_y, y_ortvalue.numpy(), rtol=1e-05, atol=1e-05)

  # After capturing, CUDA graph replay happens from this Run onwards
  session.run_with_iobinding(io_binding, ro)
  np.testing.assert_allclose(expected_y, y_ortvalue.numpy(), rtol=1e-05, atol=1e-05)

  # Update input and then replay CUDA graph with the updated input
  x_ortvalue.update_inplace(np.array([[10.0, 20.0], [30.0, 40.0], [50.0, 60.0]], dtype=np.float32))
  session.run_with_iobinding(io_binding, ro)

const auto& api = Ort::GetApi();

  struct CudaMemoryDeleter {
  explicit CudaMemoryDeleter(const Ort::Allocator* alloc) {
      alloc_ = alloc;
  }

  void operator()(void* ptr) const {
      alloc_->Free(ptr);
  }

  const Ort::Allocator* alloc_;
  };

  // Enable cuda graph in cuda provider option.
  OrtCUDAProviderOptionsV2* cuda_options = nullptr;
  api.CreateCUDAProviderOptions(&cuda_options);
  std::unique_ptr<OrtCUDAProviderOptionsV2, decltype(api.ReleaseCUDAProviderOptions)> rel_cuda_options(cuda_options, api.ReleaseCUDAProviderOptions);
  std::vector<const char*> keys{"enable_cuda_graph"};
  std::vector<const char*> values{"1"};
  api.UpdateCUDAProviderOptions(rel_cuda_options.get(), keys.data(), values.data(), 1);

  Ort::SessionOptions session_options;
  api.SessionOptionsAppendExecutionProvider_CUDA_V2(static_cast<OrtSessionOptions*>(session_options), rel_cuda_options.get();

  // Pass gpu_graph_id to RunOptions through RunConfigs
  Ort::RunOptions run_option;
  // gpu_graph_id is optional if the session uses only one cuda graph
  run_option.AddConfigEntry("gpu_graph_id", "1");

  // Create IO bound inputs and outputs.
  Ort::Session session(*ort_env, ORT_TSTR("matmul_2.onnx"), session_options);
  Ort::MemoryInfo info_cuda("Cuda", OrtAllocatorType::OrtArenaAllocator, 0, OrtMemTypeDefault);
  Ort::Allocator cuda_allocator(session, info_cuda);

  const std::array<int64_t, 2> x_shape = {3, 2};
  std::array<float, 3 * 2> x_values = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};
  auto input_data = std::unique_ptr<void, CudaMemoryDeleter>(cuda_allocator.Alloc(x_values.size() * sizeof(float)),
                                                          CudaMemoryDeleter(&cuda_allocator));
  cudaMemcpy(input_data.get(), x_values.data(), sizeof(float) * x_values.size(), cudaMemcpyHostToDevice);

  // Create an OrtValue tensor backed by data on CUDA memory
  Ort::Value bound_x = Ort::Value::CreateTensor(info_cuda, reinterpret_cast<float*>(input_data.get()), x_values.size(),
                                              x_shape.data(), x_shape.size());

  const std::array<int64_t, 2> expected_y_shape = {3, 2};
  std::array<float, 3 * 2> expected_y = {1.0f, 4.0f, 9.0f, 16.0f, 25.0f, 36.0f};
  auto output_data = std::unique_ptr<void, CudaMemoryDeleter>(cuda_allocator.Alloc(expected_y.size() * sizeof(float)),
                                                              CudaMemoryDeleter(&cuda_allocator));

  // Create an OrtValue tensor backed by data on CUDA memory
  Ort::Value bound_y = Ort::Value::CreateTensor(info_cuda, reinterpret_cast<float*>(output_data.get()),
                                              expected_y.size(), expected_y_shape.data(), expected_y_shape.size());

  Ort::IoBinding binding(session);
  binding.BindInput("X", bound_x);
  binding.BindOutput("Y", bound_y);

  // One regular run for necessary memory allocation and graph capturing
  session.Run(run_option, binding);

  // After capturing, CUDA graph replay happens from this Run onwards
  session.Run(run_option, binding);

  // Update input and then replay CUDA graph with the updated input
  x_values = {10.0f, 20.0f, 30.0f, 40.0f, 50.0f, 60.0f};
  cudaMemcpy(input_data.get(), x_values.data(), sizeof(float) * x_values.size(), cudaMemcpyHostToDevice);
  session.Run(run_option, binding);

Samples

Python

import onnxruntime as ort

model_path = '<path to model>'

providers = [
    ('CUDAExecutionProvider', {
        'device_id': 0,
        'arena_extend_strategy': 'kNextPowerOfTwo',
        'gpu_mem_limit': 2 * 1024 * 1024 * 1024,
        'cudnn_conv_algo_search': 'EXHAUSTIVE',
        'do_copy_in_default_stream': True,
    }),
    'CPUExecutionProvider',
]

session = ort.InferenceSession(model_path, providers=providers)

C/C++

Using legacy provider options struct

OrtSessionOptions* session_options = /* ... */;

OrtCUDAProviderOptions options;
options.device_id = 0;
options.arena_extend_strategy = 0;
options.gpu_mem_limit = 2 * 1024 * 1024 * 1024;
options.cudnn_conv_algo_search = OrtCudnnConvAlgoSearchExhaustive;
options.do_copy_in_default_stream = 1;

SessionOptionsAppendExecutionProvider_CUDA(session_options, &options);

Using V2 provider options struct

OrtCUDAProviderOptionsV2* cuda_options = nullptr;
CreateCUDAProviderOptions(&cuda_options);

std::vector<const char*> keys{"device_id", "gpu_mem_limit", "arena_extend_strategy", "cudnn_conv_algo_search", "do_copy_in_default_stream", "cudnn_conv_use_max_workspace", "cudnn_conv1d_pad_to_nc1d"};
std::vector<const char*> values{"0", "2147483648", "kSameAsRequested", "DEFAULT", "1", "1", "1"};

UpdateCUDAProviderOptions(cuda_options, keys.data(), values.data(), keys.size());

cudaStream_t cuda_stream;
cudaStreamCreate(&cuda_stream);
// this implicitly sets "has_user_compute_stream"
UpdateCUDAProviderOptionsWithValue(cuda_options, "user_compute_stream", cuda_stream);
OrtSessionOptions* session_options = /* ... */;
SessionOptionsAppendExecutionProvider_CUDA_V2(session_options, cuda_options);

// Finally, don't forget to release the provider options
ReleaseCUDAProviderOptions(cuda_options);

C#

var cudaProviderOptions = new OrtCUDAProviderOptions(); // Dispose this finally

var providerOptionsDict = new Dictionary<string, string>();
providerOptionsDict["device_id"] = "0";
providerOptionsDict["gpu_mem_limit"] = "2147483648";
providerOptionsDict["arena_extend_strategy"] = "kSameAsRequested";
providerOptionsDict["cudnn_conv_algo_search"] = "DEFAULT";
providerOptionsDict["do_copy_in_default_stream"] = "1";
providerOptionsDict["cudnn_conv_use_max_workspace"] = "1";
providerOptionsDict["cudnn_conv1d_pad_to_nc1d"] = "1";

cudaProviderOptions.UpdateOptions(providerOptionsDict);

SessionOptions options = SessionOptions.MakeSessionOptionWithCudaProvider(cudaProviderOptions);  // Dispose this finally

Also see the tutorial here on how to configure CUDA for C# on Windows.

Java

OrtCUDAProviderOptions cudaProviderOptions = new OrtCUDAProviderOptions(/*device id*/0); // Must be closed after the session closes

cudaProviderOptions.add("gpu_mem_limit","2147483648");
cudaProviderOptions.add("arena_extend_strategy","kSameAsRequested");
cudaProviderOptions.add("cudnn_conv_algo_search","DEFAULT");
cudaProviderOptions.add("do_copy_in_default_stream","1");
cudaProviderOptions.add("cudnn_conv_use_max_workspace","1");
cudaProviderOptions.add("cudnn_conv1d_pad_to_nc1d","1");

OrtSession.SessionOptions options = new OrtSession.SessionOptions(); // Must be closed after the session closes
options.addCUDA(cudaProviderOptions);

For documentation questions, please file an issue.

Edit this page on GitHub