CUDA Shared Memory Capacity

Introduction

CUDA shared memory is an extremely powerful feature for CUDA kernel implementation and optimization. Because CUDA shared memory is located on chip, its memory bandwidth is much larger than the global memory which is located off chip. Therefore, CUDA kernel optimization by caching memory access on shared memory can improve the performance of some operations significantly, especially for those memory-bound operations.

However, CUDA shared memory has size limits for each thread block which is 48 KB by default. Sometimes, we would like to use a little bit more shared memory for our implementations. In this blog post, I would like to discuss how to allocate static shared memory, dynamic shared memory, and how to request more than 48 KB dynamic shared memory.

Stencil Kernel

We have implemented a stencil kernel for demonstrating the allocation of CUDA shared memory. Stencil is almost mathematically equivalent as a special case of convolution whose weights are exactly 1 with valid padding.

For example, given an 1D array of $\{1, 1, 1, 1, 1, 1, 1\}$ and a stencil kernel with a radius of $2$, we will have the output 1D array $\{1, 1, 5, 5, 5, 1, 1\}$.

The stencil operation will have many redundant memory reads from the input tensor and thus is a memory-bound operation. If the memory reads are not cached and the program reads from the global memory, the performance will be poor. Therefore, we will take advantage of shared memory which is on chip to cache the memory reads and improve the performance.

Static Shared Memory

In this implementation, we allocated static shared memory whose size must be known at compile time. The implementation also supports arbitrary “valid” array size, radius, and CUDA thread block size. Also notice that when we implement the kernel, we have to pay special attention to the scenario when the radius is larger than the CUDA thread block size and the “valid” array size is not divisible by the CUDA thread block size, as it is not easy to implement them correctly.

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149

#include <cassert>#include <iostream>#include <vector>#define CHECK_CUDA_ERROR(val) check((val), #val, __FILE__, __LINE__)void check(cudaError_t err, char const* const func, char const* const file,           int const line){    if (err != cudaSuccess)    {        std::cerr << "CUDA Runtime Error at: " << file << ":" << line                  << std::endl;        std::cerr << cudaGetErrorString(err) << " " << func << std::endl;        std::exit(EXIT_FAILURE);    }}#define CHECK_LAST_CUDA_ERROR() checkLast(__FILE__, __LINE__)void checkLast(char const* const file, int const line){    cudaError_t const err{cudaGetLastError()};    if (err != cudaSuccess)    {        std::cerr << "CUDA Runtime Error at: " << file << ":" << line                  << std::endl;        std::cerr << cudaGetErrorString(err) << std::endl;        std::exit(EXIT_FAILURE);    }}template <int BLOCK_SIZE = 1024, int RADIUS = 5>__global__ void stencil_1d_kernel(int const* d_in, int* d_out,                                  int valid_array_size){    __shared__ int temp[BLOCK_SIZE + 2 * RADIUS];    // This has to be int because we will use negative indices.    int const gindex{static_cast<int>(threadIdx.x + blockIdx.x * blockDim.x)};    int const lindex{static_cast<int>(threadIdx.x) + RADIUS};    int const valid_block_size{        min(BLOCK_SIZE,            valid_array_size - static_cast<int>(blockIdx.x * blockDim.x))};    // Read input elements into shared memory    if (gindex < valid_array_size)    {        temp[lindex] = d_in[gindex];        if (RADIUS <= valid_block_size)        {            if (threadIdx.x < RADIUS)            {                temp[lindex - RADIUS] = d_in[gindex - RADIUS];                temp[lindex + valid_block_size] =                    d_in[gindex + valid_block_size];            }        }        else        {            for (int i{0}; i < RADIUS; i += valid_block_size)            {                // Some threads might have to do one more job than other                // threads.                if (lindex - RADIUS + i < RADIUS)                {                    temp[lindex - RADIUS + i] = d_in[gindex - RADIUS + i];                    temp[lindex + valid_block_size + i] =                        d_in[gindex + valid_block_size + i];                }            }        }    }    // Synchronize (ensure all the data is available)    __syncthreads();    if (gindex >= valid_array_size)    {        return;    }    // Apply the stencil    int result{0};    for (int offset{-RADIUS}; offset <= RADIUS; offset++)    {        result += temp[lindex + offset];    }    // Store the result    d_out[gindex] = result;}void stencil_1d_cpu(int const* h_in, int* h_out, int radius,                    int valid_array_size){    for (int i{0}; i < valid_array_size; ++i)    {        int result{0};        for (int offset{-radius}; offset <= radius; offset++)        {            result += h_in[i + offset];        }        h_out[i] = result;    }}int main(int argc, char** argv){    constexpr int const valid_array_size{1024 * 100 + 1};    constexpr int const block_size{1024};    constexpr int const grid_size{(valid_array_size + block_size - 1) /                                  block_size};    constexpr int const radius{1025};    int const array_size{valid_array_size + 2 * radius};    std::vector<int> const h_in(array_size, 1);    std::vector<int> h_out{h_in};    std::vector<int> h_out_reference{h_in};    stencil_1d_cpu(h_in.data() + radius, h_out_reference.data() + radius,                   radius, valid_array_size);    int* d_in;    int* d_out;    CHECK_CUDA_ERROR(cudaMalloc(&d_in, array_size * sizeof(int)));    CHECK_CUDA_ERROR(cudaMalloc(&d_out, array_size * sizeof(int)));    CHECK_CUDA_ERROR(cudaMemcpy(d_in, h_in.data(), array_size * sizeof(int),                                cudaMemcpyHostToDevice));    CHECK_CUDA_ERROR(cudaMemcpy(d_out, h_out.data(), array_size * sizeof(int),                                cudaMemcpyHostToDevice));    stencil_1d_kernel<block_size, radius><<<grid_size, block_size>>>(        d_in + radius, d_out + radius, valid_array_size);    CHECK_LAST_CUDA_ERROR();    CHECK_CUDA_ERROR(cudaDeviceSynchronize());    CHECK_CUDA_ERROR(cudaMemcpy(h_out.data(), d_out, array_size * sizeof(int),                                cudaMemcpyDeviceToHost));    for (int i{0}; i < h_out_reference.size(); ++i)    {        assert(h_out[i] == h_out_reference[i]);    }    CHECK_CUDA_ERROR(cudaFree(d_in));    CHECK_CUDA_ERROR(cudaFree(d_out));}

12

$ nvcc stencil_static_shared_memory.cu -o stencil_static_shared_memory$ ./stencil_static_shared_memory

If we increase the radius from 1025 to some larger values such as 6000, we will get the following compilation error.

12

$ nvcc stencil_static_shared_memory.cu -o stencil_static_shared_memoryptxas error   : Entry function '_Z17stencil_1d_kernelILi1024ELi6000EEvPKiPii' uses too much shared data (0xcb80 bytes, 0xc000 max)

This is because the user could only allocate the CUDA static shared memory up to 48 KB. In our use case, BLOCK_SIZE + 2 * RADIUS = $1024 + 2 \times 6000$ = $13024$ and the size of an int is $4$ bytes, therefore, the shared memory required is $17024 \times 4 / 1024$ = $50.875$ KB, which is larger than the maximum static shared memory we could have.

Dynamic Shared Memory

To use shared memory larger than 48 KB, we will have to use dynamic shared memory and it is architecture specific. Specifically, CUDA Runtime API cudaFuncSetAttribute has to be called in addition to specifying the dynamic shared memory size we want to request in the third argument in <<<...>>> for CUDA launch, and we should always check its return as it can fail during runtime on certain architectures.

The platform GPU is NVIDIA RTX 2080TI. According to the CUDA C Programming Guide, compute capability 7.x devices allow a single thread block to dynamically allocate shared memory up to 64 KB on Turing. So we could run the stencil program with a radius of 6000 on NVIDIA RTX 2080TI.

This implementation with dynamic shared memory is almost the same as the one with static shared memory.

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154

#include <cassert>#include <iostream>#include <vector>#define CHECK_CUDA_ERROR(val) check((val), #val, __FILE__, __LINE__)void check(cudaError_t err, char const* const func, char const* const file,           int const line){    if (err != cudaSuccess)    {        std::cerr << "CUDA Runtime Error at: " << file << ":" << line                  << std::endl;        std::cerr << cudaGetErrorString(err) << " " << func << std::endl;        std::exit(EXIT_FAILURE);    }}#define CHECK_LAST_CUDA_ERROR() checkLast(__FILE__, __LINE__)void checkLast(char const* const file, int const line){    cudaError_t const err{cudaGetLastError()};    if (err != cudaSuccess)    {        std::cerr << "CUDA Runtime Error at: " << file << ":" << line                  << std::endl;        std::cerr << cudaGetErrorString(err) << std::endl;        std::exit(EXIT_FAILURE);    }}template <int BLOCK_SIZE = 1024, int RADIUS = 5>__global__ void stencil_1d_kernel(int const* d_in, int* d_out,                                  int valid_array_size){    extern __shared__ int temp[];    // This has to be int because we will use negative indices.    int const gindex{static_cast<int>(threadIdx.x + blockIdx.x * blockDim.x)};    int const lindex{static_cast<int>(threadIdx.x) + RADIUS};    int const valid_block_size{        min(BLOCK_SIZE,            valid_array_size - static_cast<int>(blockIdx.x * blockDim.x))};    // Read input elements into shared memory    if (gindex < valid_array_size)    {        temp[lindex] = d_in[gindex];        if (RADIUS <= valid_block_size)        {            if (threadIdx.x < RADIUS)            {                temp[lindex - RADIUS] = d_in[gindex - RADIUS];                temp[lindex + valid_block_size] =                    d_in[gindex + valid_block_size];            }        }        else        {            for (int i{0}; i < RADIUS; i += valid_block_size)            {                // Some threads might have to do one more job than other                // threads.                if (lindex - RADIUS + i < RADIUS)                {                    temp[lindex - RADIUS + i] = d_in[gindex - RADIUS + i];                    temp[lindex + valid_block_size + i] =                        d_in[gindex + valid_block_size + i];                }            }        }    }    // Synchronize (ensure all the data is available)    __syncthreads();    if (gindex >= valid_array_size)    {        return;    }    // Apply the stencil    int result{0};    for (int offset{-RADIUS}; offset <= RADIUS; offset++)    {        result += temp[lindex + offset];    }    // Store the result    d_out[gindex] = result;}void stencil_1d_cpu(int const* h_in, int* h_out, int radius,                    int valid_array_size){    for (int i{0}; i < valid_array_size; ++i)    {        int result{0};        for (int offset{-radius}; offset <= radius; offset++)        {            result += h_in[i + offset];        }        h_out[i] = result;    }}int main(int argc, char** argv){    constexpr int const valid_array_size{1024 * 100 + 1};    constexpr int const block_size{1024};    constexpr int const grid_size{(valid_array_size + block_size - 1) /                                  block_size};    constexpr int const radius{6000};    int const array_size{valid_array_size + 2 * radius};    std::vector<int> const h_in(array_size, 1);    std::vector<int> h_out{h_in};    std::vector<int> h_out_reference{h_in};    stencil_1d_cpu(h_in.data() + radius, h_out_reference.data() + radius,                   radius, valid_array_size);    int* d_in;    int* d_out;    CHECK_CUDA_ERROR(cudaMalloc(&d_in, array_size * sizeof(int)));    CHECK_CUDA_ERROR(cudaMalloc(&d_out, array_size * sizeof(int)));    CHECK_CUDA_ERROR(cudaMemcpy(d_in, h_in.data(), array_size * sizeof(int),                                cudaMemcpyHostToDevice));    CHECK_CUDA_ERROR(cudaMemcpy(d_out, h_out.data(), array_size * sizeof(int),                                cudaMemcpyHostToDevice));    int const shared_memory_bytes{(block_size + radius * 2) * sizeof(int)};    CHECK_CUDA_ERROR(cudaFuncSetAttribute(        stencil_1d_kernel<block_size, radius>,        cudaFuncAttributeMaxDynamicSharedMemorySize, shared_memory_bytes));    stencil_1d_kernel<block_size, radius>        <<<grid_size, block_size, shared_memory_bytes>>>(            d_in + radius, d_out + radius, valid_array_size);    CHECK_LAST_CUDA_ERROR();    CHECK_CUDA_ERROR(cudaDeviceSynchronize());    CHECK_CUDA_ERROR(cudaMemcpy(h_out.data(), d_out, array_size * sizeof(int),                                cudaMemcpyDeviceToHost));    for (int i{0}; i < h_out_reference.size(); ++i)    {        assert(h_out[i] == h_out_reference[i]);    }    CHECK_CUDA_ERROR(cudaFree(d_in));    CHECK_CUDA_ERROR(cudaFree(d_out));}

12

$ nvcc stencil_dynamic_shared_memory.cu -o stencil_dynamic_shared_memory --gpu-architecture=compute_75 --gpu-code=sm_75$ ./stencil_dynamic_shared_memory

Conclusion

The reason why large shared memory can only be allocated for dynamic shared memory is that not all the GPU architecture can support certain size of shared memory that is larger than 48 KB. If static shared memory larger than 48 KB is allowed, the CUDA program will compile but fail on some specific GPU architectures, which is not desired. Therefore, to use shared memory that is larger than 48 KB, it has to be requested via dynamic shared memory during runtime. If the GPU architecture does not support shared memory of certain size, a CUDA runtime error will be returned.

References

CUDA Shared Memory Capacity

https://leimao.github.io/blog/CUDA-Shared-Memory-Capacity/

Author

Lei Mao

Posted on

07-04-2022

Updated on

12-26-2023

Licensed under

Like this article? Support the author with

Comments

Lei Mao

Artificial Intelligence
Machine Learning
Computer Science

Santa Clara, California

Posts

1008

Categories

7

Tags

662

Advertisement

Catalogue

© 2017-2025 Lei Mao  Powered by Hexo & IcarusSite UV:  Site PV:


Kernel Tuner

Guides

Features

Reference

Design documentation¶

This section provides detailed information about the design and internals
of the Kernel Tuner. This information is mostly relevant for developers.

The Kernel Tuner is designed to be extensible and support
different search and execution strategies. The current architecture of
the Kernel Tuner can be seen as:

At the top we have the kernel code and the Python script that tunes it,
which uses any of the main functions exposed in the user interface.

The strategies are responsible for iterating over and searching through
the search space. The default strategy is brute_force, which
iterates over all valid kernel configurations in the search space.
random_sample simply takes a random sample of the search space. More
advanced strategies are continuously being implemented and improved in
Kernel Tuner. The full list of supported strategies and how to use these
is explained in the API Documentation, see the options strategy and
strategy_options.

The runners are responsible for compiling and benchmarking the kernel
configurations selected by the strategy. The sequential runner is currently
the only supported runner, which does exactly what its name says. It compiles
and benchmarks configurations using a single sequential Python process.
Other runners are foreseen in future releases.

The runners are implemented on top of the core, which implements a
high-level Device Interface,
which wraps all the functionality for compiling and benchmarking
kernel configurations based on the low-level Device Function Interface.
Currently, we have
five different implementations of the device function interface, which
basically abstracts the different backends into a set of simple
functions such as ready_argument_list which allocates GPU memory and
moves data to the GPU, and functions like compile, benchmark, or
run_kernel. The functions in the core are basically the main
building blocks for implementing runners.

The observers are explained in Observers.

At the bottom, the backends are shown.
PyCUDA, CuPy, cuda-python, PyOpenCL and PyHIP are for tuning either CUDA, OpenCL, or HIP kernels.
The CompilerFunctions implementation can call any compiler, typically NVCC
or GCC is used. There is limited support for tuning Fortran kernels.
This backend was created not just to be able to tune C
functions, but in particular to tune C functions that in turn launch GPU kernels.

The rest of this section contains the API documentation of the modules
discussed above. For the documentation of the user API see the
API Documentation.

Strategies¶

Strategies are explained in Optimization strategies.

Many of the strategies use helper functions that are collected in kernel_tuner.strategies.common.

kernel_tuner.strategies.common¶

Get the strategy-specific options or their defaults from user-supplied strategy_options.

Generate docstring for a ‘tune’ method of a strategy.

Generate documentation for the supported strategy options and their defaults.

Helper func to do the inverse of the ‘unscale’ function.

Prepare method specific arguments.

Prepare method specific options.

Helper func that for each param selects the closest actual value.

Helper func that snaps a scaled variable to the nearest config.

Runners¶

kernel_tuner.runners.sequential.SequentialRunner¶

SequentialRunner is used for tuning with a single process/thread.

Instantiate the SequentialRunner.

kernel_source (kernel_tuner.core.KernelSource) – The kernel source

kernel_options (kernel_tuner.interface.Options) – A dictionary with all options for the kernel.

device_options (kernel_tuner.interface.Options) – A dictionary with all options for the device
on which the kernel should be tuned.

iterations (int) – The number of iterations used for benchmarking
each kernel instance.

Iterate through the entire parameter space using a single Python process.

parameter_space (iterable) – The parameter space as an iterable.

tuning_options (kernel_tuner.iterface.Options) – A dictionary with all options regarding the tuning
process.

A list of dictionaries for executed kernel configurations and their
execution times.

dict())

kernel_tuner.runners.sequential.SimulationRunner¶

SimulationRunner is used for tuning with a single process/thread.

Instantiate the SimulationRunner.

kernel_source (kernel_tuner.core.KernelSource) – The kernel source

kernel_options (kernel_tuner.interface.Options) – A dictionary with all options for the kernel.

device_options (kernel_tuner.interface.Options) – A dictionary with all options for the device
on which the kernel should be tuned.

iterations (int) – The number of iterations used for benchmarking
each kernel instance.

Iterate through the entire parameter space using a single Python process.

parameter_space (iterable) – The parameter space as an iterable.

tuning_options (kernel_tuner.iterface.Options) – A dictionary with all options regarding the tuning
process.

A list of dictionaries for executed kernel configurations and their
execution times.

dict()

Device Interfaces¶

kernel_tuner.core.DeviceInterface¶

Class that offers a High-Level Device Interface to the rest of the Kernel Tuner

Instantiate the DeviceInterface, based on language in kernel source

kernel_source (kernel_tuner.core.KernelSource) – The kernel sources

device (int) – CUDA/OpenCL device to use, in case you have multiple
CUDA-capable GPUs or OpenCL devices you may use this to select one,
0 by default. Ignored if you are tuning host code by passing lang=”C”.

platform – OpenCL platform to use, in case you have multiple
OpenCL platforms you may use this to select one,
0 by default. Ignored if not using OpenCL.

lang (string) – Specifies the language used for GPU kernels.
Currently supported: “CUDA”, “OpenCL”, “HIP” or “C”

compiler_options (list of strings) – The compiler options to use when compiling kernels for this device.

iterations (int) – Number of iterations to be used when benchmarking using this device.

times (bool) – Return the execution time of all iterations.

benchmark the kernel instance

Benchmark continuously for at least ‘duration’ seconds

Benchmark one kernel execution at a time

runs the kernel once and checks the result against answer

compile the kernel for this specific instance

adds constant memory arguments to the most recently compiled module

adds shared memory arguments to the most recently compiled module

adds texture memory arguments to the most recently compiled module

create kernel instance from kernel source, parameters, problem size, grid divisors, and so on

Return dictionary with information about the environment

perform a device to host memory copy

Get a flat list of arguments based on the configuration given by params

ready argument list to be passed to the kernel, allocates gpu mem if necessary

Run a compiled kernel instance on a device

kernel_tuner.backends.pycuda.PyCudaFunctions¶

Class that groups the CUDA functions on maintains state about the device.

Instantiate PyCudaFunctions object used for interacting with the CUDA device.

Instantiating this object will inspect and store certain device properties at
runtime, which are used during compilation and/or execution of kernels by the
kernel tuner. It also maintains a reference to the most recently compiled
source module for copying data to constant memory before kernel launch.

device (int) – Number of CUDA device to use for this context

iterations (int) – Number of iterations used while benchmarking a kernel, 7 by default.

Call the CUDA compiler to compile the kernel, return the device function.

kernel_name (string) – The name of the kernel to be compiled, used to lookup the
function after compilation.

kernel_string (string) – The CUDA kernel code that contains the function kernel_name

An CUDA kernel that can be called directly.

pycuda.driver.Function

Adds constant memory arguments to the most recently compiled module.

cmem_args (dict( string: numpy.ndarray, ... )) – A dictionary containing the data to be passed to the
device constant memory. The format to be used is as follows: A
string key is used to name the constant memory symbol to which the
value needs to be copied. Similar to regular arguments, these need
to be numpy objects, such as numpy.ndarray or numpy.int32, and so on.

Add shared memory arguments to the kernel.

Adds texture memory arguments to the most recently compiled module.

texmem_args (dict) – A dictionary containing the data to be passed to the
device texture memory. See tune_kernel().

Returns True if the kernel has finished, False otherwise.

Perform a device to host memory copy.

dest (numpy.ndarray) – A numpy array in host memory to store the data

src (pycuda.driver.DeviceAllocation) – A GPU memory allocation unit

Perform a host to device memory copy.

dest (pycuda.driver.DeviceAllocation) – A GPU memory allocation unit

src (numpy.ndarray) – A numpy array in host memory to store the data

Set the memory in allocation to the value in value.

allocation (pycuda.driver.DeviceAllocation) – A GPU memory allocation unit

value (a single 8-bit unsigned int) – The value to set the memory to

size (int) – The size of to the allocation unit in bytes

Ready argument list to be passed to the kernel, allocates gpu mem.

arguments (list(numpy objects)) – List of arguments to be passed to the kernel.
The order should match the argument list on the CUDA kernel.
Allowed values are numpy.ndarray, and/or numpy.int32, numpy.float32, and so on.

A list of arguments that can be passed to an CUDA kernel.

list( pycuda.driver.DeviceAllocation, numpy.int32, … )

Runs the CUDA kernel passed as ‘func’.

func (pycuda.driver.Function) – A PyCuda kernel compiled for this specific kernel configuration

gpu_args (list( pycuda.driver.DeviceAllocation, numpy.int32, ...)) – A list of arguments to the kernel, order should match the
order in the code. Allowed values are either variables in global memory
or single values passed by value.

threads (tuple(int, int, int)) – A tuple listing the number of threads in each dimension of
the thread block

grid (tuple(int, int)) – A tuple listing the number of thread blocks in each dimension
of the grid

Records the event that marks the start of a measurement.

Records the event that marks the end of a measurement.

Halts execution until device has finished its tasks.

kernel_tuner.backends.cupy.CupyFunctions¶

Class that groups the Cupy functions on maintains state about the device.

Instantiate CupyFunctions object used for interacting with the CUDA device.

Instantiating this object will inspect and store certain device properties at
runtime, which are used during compilation and/or execution of kernels by the
kernel tuner. It also maintains a reference to the most recently compiled
source module for copying data to constant memory before kernel launch.

device (int) – Number of CUDA device to use for this context

iterations (int) – Number of iterations used while benchmarking a kernel, 7 by default.

Call the CUDA compiler to compile the kernel, return the device function.

kernel_name (string) – The name of the kernel to be compiled, used to lookup the
function after compilation.

kernel_string (string) – The CUDA kernel code that contains the function kernel_name

An CUDA kernel that can be called directly.

cupy.RawKernel

Adds constant memory arguments to the most recently compiled module.

cmem_args (dict( string: numpy.ndarray, ... )) – A dictionary containing the data to be passed to the
device constant memory. The format to be used is as follows: A
string key is used to name the constant memory symbol to which the
value needs to be copied. Similar to regular arguments, these need
to be numpy objects, such as numpy.ndarray or numpy.int32, and so on.

Add shared memory arguments to the kernel.

Adds texture memory arguments to the most recently compiled module.

texmem_args (dict) – A dictionary containing the data to be passed to the
device texture memory. See tune_kernel().

Returns True if the kernel has finished, False otherwise.

Perform a device to host memory copy.

dest (numpy.ndarray) – A numpy array in host memory to store the data

src (cupy.ndarray) – A GPU memory allocation unit

Perform a host to device memory copy.

dest (cupy.ndarray) – A GPU memory allocation unit

src (numpy.ndarray) – A numpy array in host memory to store the data

Set the memory in allocation to the value in value.

allocation (cupy.ndarray) – A GPU memory allocation unit

value (a single 8-bit unsigned int) – The value to set the memory to

size (int) – The size of to the allocation unit in bytes

Ready argument list to be passed to the kernel, allocates gpu mem.

arguments (list(numpy objects)) – List of arguments to be passed to the kernel.
The order should match the argument list on the CUDA kernel.
Allowed values are numpy.ndarray, and/or numpy.int32, numpy.float32, and so on.

A list of arguments that can be passed to an CUDA kernel.

list( cupy.ndarray, numpy.int32, … )

Runs the CUDA kernel passed as ‘func’.

func (cupy.RawKernel) – A cupy kernel compiled for this specific kernel configuration

gpu_args (list( cupy.ndarray, numpy.int32, ...)) – A list of arguments to the kernel, order should match the
order in the code. Allowed values are either variables in global memory
or single values passed by value.

threads (tuple(int, int, int)) – A tuple listing the number of threads in each dimension of
the thread block

grid (tuple(int, int)) – A tuple listing the number of thread blocks in each dimension
of the grid

Records the event that marks the start of a measurement.

Records the event that marks the end of a measurement.

Halts execution until device has finished its tasks.

kernel_tuner.backends.nvcuda.CudaFunctions¶

Class that groups the Cuda functions on maintains state about the device.

Instantiate CudaFunctions object used for interacting with the CUDA device.

Instantiating this object will inspect and store certain device properties at
runtime, which are used during compilation and/or execution of kernels by the
kernel tuner. It also maintains a reference to the most recently compiled
source module for copying data to constant memory before kernel launch.

device (int) – Number of CUDA device to use for this context

iterations (int) – Number of iterations used while benchmarking a kernel, 7 by default.

compiler_options – Compiler options for the CUDA runtime compiler

observers – List of Observer type objects

Call the CUDA compiler to compile the kernel, return the device function.

kernel_name (string) – The name of the kernel to be compiled, used to lookup the
function after compilation.

kernel_string (string) – The CUDA kernel code that contains the function kernel_name

A kernel that can be launched by the CUDA runtime

Adds constant memory arguments to the most recently compiled module.

cmem_args (dict( string: numpy.ndarray, ... )) – A dictionary containing the data to be passed to the
device constant memory. The format to be used is as follows: A
string key is used to name the constant memory symbol to which the
value needs to be copied. Similar to regular arguments, these need
to be numpy objects, such as numpy.ndarray or numpy.int32, and so on.

Add shared memory arguments to the kernel.

Adds texture memory arguments to the most recently compiled module.

texmem_args (dict) – A dictionary containing the data to be passed to the
device texture memory. See tune_kernel().

Returns True if the kernel has finished, False otherwise.

Perform a device to host memory copy.

dest (numpy.ndarray) – A numpy array in host memory to store the data

src (cuda.CUdeviceptr) – A GPU memory allocation unit

Perform a host to device memory copy.

dest (cuda.CUdeviceptr) – A GPU memory allocation unit

src (numpy.ndarray) – A numpy array in host memory to store the data

Set the memory in allocation to the value in value.

allocation (cupy.ndarray) – A GPU memory allocation unit

value (a single 8-bit unsigned int) – The value to set the memory to

size (int) – The size of to the allocation unit in bytes

Ready argument list to be passed to the kernel, allocates gpu mem.

arguments (list(numpy objects)) – List of arguments to be passed to the kernel.
The order should match the argument list on the CUDA kernel.
Allowed values are numpy.ndarray, and/or numpy.int32, numpy.float32, and so on.

A list of arguments that can be passed to an CUDA kernel.

list( pycuda.driver.DeviceAllocation, numpy.int32, … )

Runs the CUDA kernel passed as ‘func’.

func (cuda.CUfunction) – A CUDA kernel compiled for this specific kernel configuration

gpu_args (list( cupy.ndarray, numpy.int32, ...)) – A list of arguments to the kernel, order should match the
order in the code. Allowed values are either variables in global memory
or single values passed by value.

threads (tuple(int, int, int)) – A tuple listing the number of threads in each dimension of
the thread block

grid (tuple(int, int)) – A tuple listing the number of thread blocks in each dimension
of the grid

Records the event that marks the start of a measurement.

Records the event that marks the end of a measurement.

Halts execution until device has finished its tasks.

kernel_tuner.backends.opencl.OpenCLFunctions¶

Class that groups the OpenCL functions on maintains some state about the device.

Creates OpenCL device context and reads device properties.

device (int) – The ID of the OpenCL device to use for benchmarking

iterations (int) – The number of iterations to run the kernel during benchmarking, 7 by default.

Call the OpenCL compiler to compile the kernel, return the device function.

kernel_name (string) – The name of the kernel to be compiled, used to lookup the
function after compilation.

kernel_string (string) – The OpenCL kernel code that contains the function kernel_name

An OpenCL kernel that can be called directly.

pyopencl.Kernel

This method must implement the allocation and copy of constant memory to the GPU.

This method must implement the dynamic allocation of shared memory on the GPU.

This method must implement the allocation and copy of texture memory to the GPU.

Returns True if the kernel has finished, False otherwise.

Perform a device to host memory copy.

dest (numpy.ndarray) – A numpy array in host memory to store the data

src (pyopencl.Buffer) – An OpenCL Buffer to copy data from

Perform a host to device memory copy.

dest (pyopencl.Buffer) – An OpenCL Buffer to copy data from

src (numpy.ndarray) – A numpy array in host memory to store the data

Set the memory in allocation to the value in value.

allocation (pyopencl.Buffer) – An OpenCL Buffer to fill

value (a single 32-bit int) – The value to set the memory to

size (int) – The size of to the allocation unit in bytes

Ready argument list to be passed to the kernel, allocates gpu mem.

arguments (list(numpy objects)) – List of arguments to be passed to the kernel.
The order should match the argument list on the OpenCL kernel.
Allowed values are numpy.ndarray, and/or numpy.int32, numpy.float32, and so on.

A list of arguments that can be passed to an OpenCL kernel.

list( pyopencl.Buffer, numpy.int32, … )

Runs the OpenCL kernel passed as ‘func’.

func (pyopencl.Kernel) – An OpenCL Kernel

gpu_args (list( pyopencl.Buffer, numpy.int32, ...)) – A list of arguments to the kernel, order should match the
order in the code. Allowed values are either variables in global memory
or single values passed by value.

threads (tuple(int, int, int)) – A tuple listing the number of work items in each dimension of
the work group.

grid (tuple(int, int)) – A tuple listing the number of work groups in each dimension
of the NDRange.

Records the event that marks the start of a measurement.

In OpenCL the event is created when the kernel is launched

Records the event that marks the end of a measurement.

In OpenCL the event is created when the kernel is launched

Halts execution until device has finished its tasks.

kernel_tuner.backends.compiler.CompilerFunctions¶

Class that groups the code for running and compiling C functions

instantiate CFunctions object used for interacting with C code

iterations (int) – Number of iterations used while benchmarking a kernel, 7 by default.

unload the previously loaded shared library

call the C compiler to compile the kernel, return the function

kernel_instance (kernel_tuner.core.KernelInstance) – An object representing the specific instance of the tunable kernel
in the parameter space.

An ctypes function that can be called directly.

ctypes._FuncPtr

Returns True if the kernel has finished, False otherwise

C backend does not support asynchronous launches

a simple memcpy copying from an Argument to a numpy array

dest (np.ndarray or cupy.ndarray) – A numpy or cupy array to store the data

src (Argument) – An Argument for some memory allocation

a simple memcpy copying from a numpy array to an Argument

dest (Argument) – An Argument for some memory allocation

src (np.ndarray or cupy.ndarray) – A numpy or cupy array containing the source data

set the memory in allocation to the value in value

allocation (Argument) – An Argument for some memory allocation unit

value (a single 8-bit unsigned int) – The value to set the memory to

size (int) – The size of to the allocation unit in bytes

ready argument list to be passed to the C function

arguments (list(numpy or cupy objects)) – List of arguments to be passed to the C function.
The order should match the argument list on the C function.
Allowed values are np.ndarray, cupy.ndarray, and/or np.int32, np.float32, and so on.

A list of arguments that can be passed to the C function.

list(Argument)

runs the kernel once, returns whatever the kernel returns

func (ctypes._FuncPtr) – A C function compiled for this specific configuration

c_args (list(Argument)) – A list of arguments to the function, order should match the
order in the code. The list should be prepared using
ready_argument_list().

threads (any) – Ignored, but left as argument for now to have the same
interface as CudaFunctions and OpenCLFunctions.

grid (any) – Ignored, but left as argument for now to have the same
interface as CudaFunctions and OpenCLFunctions.

A robust average of values returned by the C function.

float

Records the event that marks the start of a measurement

C backend does not use events

Records the event that marks the end of a measurement

C backend does not use events

Halts execution until device has finished its tasks

C backend does not support asynchronous launches

kernel_tuner.backends.hip.HipFunctions¶

Class that groups the HIP functions on maintains state about the device.

Instantiate HipFunctions object used for interacting with the HIP device.

Instantiating this object will inspect and store certain device properties at
runtime, which are used during compilation and/or execution of kernels by the
kernel tuner. It also maintains a reference to the most recently compiled
source module for copying data to constant memory before kernel launch.

device (int) – Number of HIP device to use for this context

iterations (int) – Number of iterations used while benchmarking a kernel, 7 by default.

Call the HIP compiler to compile the kernel, return the function.

kernel_instance (kernel_tuner.core.KernelInstance) – An object representing the specific instance of the tunable kernel
in the parameter space.

An ctypes function that can be called directly.

ctypes._FuncPtr

Adds constant memory arguments to the most recently compiled module.

cmem_args (dict( string: numpy.ndarray, ... )) – A dictionary containing the data to be passed to the
device constant memory. The format to be used is as follows: A
string key is used to name the constant memory symbol to which the
value needs to be copied. Similar to regular arguments, these need
to be numpy objects, such as numpy.ndarray or numpy.int32, and so on.

Add shared memory arguments to the kernel.

Copy texture memory arguments. Not yet implemented.

Returns True if the kernel has finished, False otherwise.

Perform a device to host memory copy.

dest (numpy.ndarray) – A numpy array in host memory to store the data

src (ctypes ptr) – A GPU memory allocation unit

Perform a host to device memory copy.

dest (ctypes ptr) – A GPU memory allocation unit

src (numpy.ndarray) – A numpy array in host memory to store the data

Set the memory in allocation to the value in value.

allocation (ctypes ptr) – A GPU memory allocation unit

value (a single 8-bit unsigned int) – The value to set the memory to

size (int) – The size of to the allocation unit in bytes

Ready argument list to be passed to the HIP function.

arguments (list(numpy objects)) – List of arguments to be passed to the HIP function.
The order should match the argument list on the HIP function.
Allowed values are np.ndarray, and/or np.int32, np.float32, and so on.

Ctypes structure of arguments to be passed to the HIP function.

ctypes structure

Runs the HIP kernel passed as ‘func’.

func (ctypes pionter) – A HIP kernel compiled for this specific kernel configuration

gpu_args (ctypes structure) – A ctypes structure of arguments to the kernel, order should match the
order in the code. Allowed values are either variables in global memory
or single values passed by value.

threads (tuple(int, int, int)) – A tuple listing the number of threads in each dimension of
the thread block

grid (tuple(int, int, int)) – A tuple listing the number of thread blocks in each dimension
of the grid

Records the event that marks the start of a measurement.

Records the event that marks the end of a measurement.

Halts execution until device has finished its tasks.

Util Functions¶

kernel_tuner.util¶

Module for kernel tuner utility functions.

Class we use for dumping Numpy objects to JSON.

Implement this method in a subclass such that it returns
a serializable object for o, or calls the base implementation
(to raise a TypeError).

For example, to support arbitrary iterators, you could
implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return JSONEncoder.default(self, o)

Exception used to raise when compiling or launching a kernel fails for a reason that can be expected.

Exception thrown when a stop criterion has been reached.

Raise an exception if a kernel arguments do not match host arguments.

Check if the numpy.dtype matches the type used in the code.

Check whether a specific configuration meets the search space restrictions.

Checks if max_fevals is reached or time limit is exceeded.

Check on maximum thread block dimensions.

Raise an exception if a tune parameter has a forbidden name.

Parses restrictions from a list of strings into a list of strings, Functions, or Constraints (if try_to_constraint) and parameters used, or a single Function if monolithic is true.

Combines restrictions and a check on the max thread block dimension to check config validity.

Convert the python-constraint to a function for backwards compatibility.

if cache file was not properly closed, pretend it was properly closed

Checking the status of CUDA calls using the NVIDIA cuda-python backend.

Delete a temporary file, don’t complain if no longer exists.

Attempt to detect language from the kernel_string.

Dumps a string in the cache, this omits the several checks of store_cache() to speed up the process - with great power comes great responsibility!

Returns the best configuration from a list of results according to some objective.

Return a compact string representation of a measurement.

Compute grid dims based on problem sizes and listed grid divisors.

Combine the parameters to a string mostly used for debug output use of dict is advised.

Retrieve the kernel source and return as a string.

This function processes the passed kernel_source argument, which could be
a function, a string with a filename, or just a string with code already.

If kernel_source is a function, the function is called with instance
parameters in ‘params’ as the only argument.

If kernel_source looks like filename, the file is read in, but if
the file does not exist, it is assumed that the string is not a filename
after all.

kernel_source (string or callable) – One of the sources for the kernel, could be a
function that generates the kernel code, a string containing a filename
that points to the kernel source, or just a string that contains the code.

params – Dictionary containing the tunable parameters for this specific
kernel instance, only needed when kernel_source is a generator.

A string containing the kernel code.

string

Compute current problem size.

Return a dict with kernel instance specific size.

Return a string in the form of temp_X, where X is a large integer.

Thread block size from tuning params, currently using convention.

Sum all timings and put their totals in the env.

Attempt to detect whether source code or a filename was passed.

Normalize a user-specified verify function.

The user-specified function has two required positional arguments (answer, result_host),
and an optional keyword (or keyword-only) argument atol. We normalize it to always accept
an atol keyword argument.

Undefined behaviour if the passed function does not match the required signatures.

Parses restrictions from a list of strings into compilable functions and constraints, or a single compilable function (if monolithic is True). Returns a list of tuples of (strings or constraints) and parameters.

Prepare kernel string for compilation.

Prepends the kernel with a series of C preprocessor defines specific
to this kernel instance:

the thread block dimensions

the grid dimensions

tunable parameters

kernel_name (string) – Name of the kernel.

kernel_string (string) – One of the source files of the kernel as a string containing code.

params (dict) – A dictionary containing the tunable parameters specific to this instance.

grid (tuple(x,y,z)) – A tuple with the grid dimensions for this specific instance.

threads (tuple(x,y,z)) – A tuple with the thread block dimensions for this specific instance.

block_size_names (tuple(string)) – A tuple with the names of the thread block dimensions used
in the code. By default this is [“block_size_x”, …], but the user
may supply different names if they prefer.

defines (dict or None) – A dict that describes the variables that should be defined as
preprocessor macros. Each keys should be the variable names and each value
is either a string or a function that returns a string. If None, each
tunable parameter is defined as preprocessor macro instead.

A string containing the source code made specific to this kernel instance.

string

Print the configuration string with tunable parameters and benchmark results.

Print the configuration string with tunable parameters and benchmark results.

Cache file for storing tuned configurations.

the cache file is stored using JSON and uses the following format:

{ device_name: "name of device"
  kernel_name: "name of kernel"
  problem_size: (int, int, int)
  tune_params_keys: list
  tune_params:
  cache: {
    "x1,x2,..xN": {"block_size_x": x1, ..., time=0.234342},
    "y1,y2,..yN": {"block_size_x": y1, ..., time=0.134233},
  }
}

The last two closing brackets are not required, and everything
should work as expected if these are missing. This is to allow to continue
from an earlier (abruptly ended) tuning session.

Process user-defined metrics for derived benchmark results.

Metrics must be a dictionary to support composable metrics. The dictionary keys describe
the name given to this user-defined metric and will be used as the key in the results dictionaries
return by Kernel Tuner. The values describe how to calculate the user-defined metric, using either a
string expression in which the tunable parameters and benchmark results can be used as variables, or
as a function that accepts a dictionary as argument.

Example:
metrics = dict()
metrics[“x”] = “10000 / time”
metrics[“x2”] = “x*x”

Note that the values in the metric dictionary can also be functions that accept params as argument.

Example:
metrics = dict()
metrics[“GFLOP/s”] = lambda p : 10000 / p[“time”]

params (dict) – A dictionary with tunable parameters and benchmark results.

metrics (dict) – A dictionary with user-defined metrics that can be used to create derived benchmark results.

An updated params dictionary with the derived metrics inserted along with the benchmark results.

dict

Read the cachefile into a dictionary, if open_cache=True prepare the cachefile for appending.

Return the contents of the file named filename or None if file not found.

Replace occurrences of the tuning params with their current value.

Compute problem size, thread block and grid dimensions for this kernel.

Stores a new entry (key, params) to the cachefile.

Dump the contents of string to a file called filename.

© Copyright 2016-2024, Ben van Werkhoven, Alessio Sclocco, Stijn Heldens, Floris-Jan Willemsen, Willem-Jan Palenstijn, Bram Veenboer and Richard Schoonhoven.


Instructor View

EPISODES

RESOURCES

Registers, Global, and Local Memory

Last updated on 2024-11-19 |

        Edit this page

Estimated time:  45 minutes

Overview

Questions

Objectives

Now that we know how to write a CUDA kernel to run code on the GPU,
and how to use the Python interface provided by CuPy to execute it, it
is time to look at the different memory spaces in the CUDA programming
model.

Registers

Registers are fast on-chip memories that are used to store operands
for the operations executed by the computing cores.

Did we encounter registers in the vector_add code used
in the previous episode? Yes we did! The variable item is,
in fact, stored in a register for at least part, if not all, of a
thread’s execution. In general all scalar variables defined in CUDA code
are stored in registers.

Registers are local to a thread, and each thread has exclusive access
to its own registers: values in registers cannot be accessed by other
threads, even from the same block, and are not available for the host.
Registers are also not permanent, therefore data stored in registers is
only available during the execution of a thread.

Challenge: how many registers are we using?

Can you guess how many registers are we using in the following
vector_add code?

C

extern "C"
__global__ void vector_add(const float * A, const float * B, float * C, const int size)
{
   int item = (blockIdx.x * blockDim.x) + threadIdx.x;

   if ( item < size )
   {
      C[item] = A[item] + B[item];
   }
}

Show me the solution

In general, it is not possible to exactly know how many registers the
compiler will use without examining the output generated by the compiler
itself. However, we can roughly estimate the amount of necessary
registers based on the variables used. We most probably need one
register to store the variable item, two registers to store
the content of A[item] and B[item], and one
additional register to store the sum A[item] + B[item]. So
the number of registers that vector_add probably uses is
4.

If we want to make registers use more explicit in the
vector_add code, we can try to rewrite it in a slightly
different, but equivalent, way.

C

extern "C"
__global__ void vector_add(const float * A, const float * B, float * C, const int size)
{
   int item = (blockIdx.x * blockDim.x) + threadIdx.x;
   float temp_a, temp_b, temp_c;

   if ( item < size )
   {
       temp_a = A[item];
       temp_b = B[item];
       temp_c = temp_a + temp_b;
       C[item] = temp_c;
   }
}

In this new version of vector_add we explicitly declare
three float variables to store the values loaded from
memory and the sum of our input items, making the estimation of used
registers more obvious.

This it totally unnecessary in the case of our example, because the
compiler will determine on its own the right amount of registers to
allocate per thread, and what to store in them. However, explicit
register usage can be important for reusing items already loaded from
memory.

Callout

Registers are the fastest memory on the GPU, so using them to
increase data reuse is an important performance optimization. We will
look at some examples of manually using registers to improve performance
in future episodes.

Small CUDA arrays, which size is known at compile time, will also be
allocated in registers by the compiler. We can rewrite the previous
version of vector_add to work with an array of
registers.

C

extern "C"
__global__ void vector_add(const float * A, const float * B, float * C, const int size)
{
   int item = (blockIdx.x * blockDim.x) + threadIdx.x;
   float temp[3];

   if ( item < size )
   {
       temp[0] = A[item];
       temp[1] = B[item];
       temp[2] = temp[0] + temp[1];
       C[item] = temp[2];
   }
}

Once again, this is not something that we would normally do, and it
is provided only as an example of how to work with arrays of
registers.

Global Memory

Global memory can be considered the main memory space of the GPU in
CUDA. It is allocated, and managed, by the host, and it is accessible to
both the host and the GPU, and for this reason the global memory space
can be used to exchange data between the two. It is the largest memory
space available, and therefore it can contain much more data than
registers, but it is also slower to access. This memory space does not
require any special memory space identifier.

Challenge: identify when global memory is used

Observe the code of the following vector_add and
identify where global memory is used.

C

extern "C"
__global__ void vector_add(const float * A, const float * B, float * C, const int size)
{
   int item = (blockIdx.x * blockDim.x) + threadIdx.x;

   if ( item < size )
   {
      C[item] = A[item] + B[item];
   }
}

Show me the solution

The vectors A, B, and C are
stored in global memory.

Memory allocated on the host, and passed as a parameter to a kernel,
is by default allocated in global memory.

Global memory is accessible by all threads, from all thread blocks.
This means that a thread can read and write any value in global
memory.

Callout

While global memory is visible to all threads, remember that global
memory is not coherent, and changes made by one thread block may not be
available to other thread blocks during the kernel execution. However,
all memory operations are finalized when the kernel terminates.

Local Memory

Memory can also be statically allocated from within a kernel, and
according to the CUDA programming model such memory will not be global
but local memory. Local memory is only visible, and therefore
accessible, by the thread allocating it. So all threads executing a
kernel will have their own privately allocated local memory.

Challenge: use local memory

Modify the following of vector_add so that intermediate
data products are stored in local memory, and only the final result is
saved into global memory.

C

extern "C"
__global__ void vector_add(const float * A, const float * B, float * C, const int size)
{
   int item = (blockIdx.x * blockDim.x) + threadIdx.x;

   if ( item < size )
   {
      C[item] = A[item] + B[item];
   }
}

Hint: have a look at the example using an array of registers, but
find a way to use a variable and not a constant for the size.

Show me the solution

We need to pass the size of the local array as a new parameter to the
kernel, because if we just specified 3 in the code, the
compiler would allocate registers and not local memory.

C

extern "C"
__global__ void vector_add(const float * A, const float * B, float * C, const int size, const int local_memory_size)
{
   int item = (blockIdx.x * blockDim.x) + threadIdx.x;
   float local_memory[local_memory_size];

   if ( item < size )
   {
      local_memory[0] = A[item];
      local_memory[1] = B[item];
      local_memory[2] = local_memory[0] + local_memory[1];
      C[item] = local_memory[2];
   }
}

The host code could be modified adding one line and changing the way
the kernel is called.

PYTHON

local_memory_size = 3
vector_add_gpu((2, 1, 1), (size // 2, 1, 1), (a_gpu, b_gpu, c_gpu, size, local_memory_size))

Local memory is not not a particularly fast memory, and in fact it
has similar throughput and latency of global memory, but it is much
larger than registers. As an example, local memory is automatically used
by the CUDA compiler to store spilled registers, i.e. to temporarily
store variables that cannot be kept in registers anymore because there
is not enough space in the register file, but that will be used again in
the future and so cannot be erased.

Key Points

This lesson is subject to the Code of Conduct

Edit on GitHub

        | Contributing
        | Source

Cite | Contact | About

Materials licensed under CC-BY 4.0 by the authors

Template licensed under CC-BY 4.0 by The Carpentries

Built with sandpaper (0.16.10), pegboard (0.7.7), and varnish (1.0.5)


Table of contents

Reference

Launching the GPU kernel

CUDA kernels

Now we learned how to interact with CUDA API, we can ask the GPU to execute a code.
GPU is an accelerator, which means that it was designed to be used alongside the conventional CPU.
Any code that uses GPU must have two parts: one that is executed on a CPU and one that is ported to the GPU.
CPU still controls the workflow, with GPU helping out with the more compute-intensive parts of the workflow.
This is why the CPU is normally referred to as a host, and GPU — as a device.
With this hardware structure, the API should have a means to switch from CPU to GPU execution.
This is done using special functions, called kernels.
To separate these functions from usual functions, they are marked by function specifier __global__:

__global__ void gpu_kernel(..)

What __global__ essentially means is that the function should be called from the host code, but will be executed on the device.
Since this function will be executed in many threads, the return value must be void: otherwise it would not be clear which of the threads should do the return.
The rest of the function definition is the same as with any C/C++ function: its name has the same limitations as a normal C function, it can have any number of arguments of any type, it is even can be templated.
Since the call of the kernel function happens in the host code but it is executed on the device, this place in the code marks a transition from single-thread execution to a many-thread execution.
One can think of it being a loop, each step of which is executed simultaneously.
As in loop, one needs an index, to differentiate the threads.
Here it gets a little bit complicated and we need to step back and re-iterate on how the GPUs are organized on a hardware level.

A simple example of the division of threads (green squares) in blocks (cyan rectangles).
The equally-sized blocks contain four threads each.
The thread index starts from zero in each block.
Hence the “global” thread index should be computed from the thread index, block index and block size.
This is explained for the thread #3 in block #2 (blue numbers).
The thread blocks are mapped to SMs for execution, with all threads within a block executing on the same device.
The number of threads within one block does not have to be equal to the number of execution units within multiprocessor.
In fact, GPUs can switch between software threads very efficiently, putting threads that currently wait for the data on hold and releasing the resources for threads that are ready for computations.
For efficient GPU utilization, the number of threads per block has to be couple of factors higher than the number of computing units on the multiprocessor.
Same is true for the number of thread blocks, which can and should be higher than the number of available multiprocessor in order to use the GPU computational resources efficiently.

The GPU contains several Streaming Modules (SMs, or multiprocessors), each with many compute units (see Figure above).
Every compute unit can execute commands.
So the entire GPU is first divided into streaming modules (or multiprocessors) and each multiprocessor contains many execution units.
To reflect this hierarchy on a software level, threads are grouped in identically sized blocks.
Each block is assigned into a streaming module for execution.
This collection of the thread blocks is usually called “grid”, which also can be multi-dimensional.

Although it may seem a bit complicated at the beginning, the grouping of threads open extra opportunities for synchronization and data exchange.
Since threads in a block are executed on a same SM, they can shared the data and can do fast communications.
This can be leveraged when designing and optimizing the code for GPU execution, and we will touch this topic later.

Given that the threads on a GPU are organized in a hierarchical manner, the global index of a thread should be computed from its in-block index, the index of execution block and the execution block size.
To get the global thread index, one can start the kernel function with:

__global__ void gpu_kernel(..)
{
   int i = threadIdx.x + blockIdx.x*blockDim.x;
}

Here, threadIdx.x, blockIdx.x and blockDim.x are internal variables that are always available inside the device function.
They are, respectively, index of thread in a block, index of the block and the size of the block.

Here, we use one-dimensional arrangement of blocks and threads (hence, the .x).
More on multi-dimensional grids and CUDA built-in simple types later, for now we assume that the rest of the components equal to 1.
Since the index i is unique for each thread in an entire grid, it is usually called “global” index.
Global index can than be used to identify the GPU thread and assign a data elements to it.
For example, if we are applying the same function on different data elements in an array, we can use the global index to identify the element of this array for a particular thread.
In a CPU code, this would normally be done in a loop over all consecutive values in an array.
In a GPU code, we assign a thread to each element of the array.

Now the kernel is defined, we can call it from the host code.
Since the kernel will be executed in a grid of threads, so the kernel launch should be supplied with the configuration of the grid.
In CUDA this is done by adding kernel cofiguration, <<<numBlocks, threadsPerBlock>>>, to the function call:

gpu_kernel<<<numBlocks, threadsPerBlock>>>(..)

Here, numBlocks is the total number of thread blocks in the grid, threadsPerBlock is the number of threads in a single block.
Note, that these values can be integers, or can be two-dimensional of three-dimensional vectors, if this is more suitable for the kernel.
More on that later.
In case of one-dimensional grid, the kernel configuration can be specified by two integer values.
The threadsPerBlock can be arbitrary chosen.
It should be larger that the number of CUDA cores in the SM to fully occupy the device, but lower than the limit of 1024 (see the technical specifications).
Values of 256 or 512 are frequently used.
The total number of threads that will be created is the multiple of numBlocks and threadsPerBlock.

Kernels are asynchronous

In CUDA, the execution of the kernel is asynchronous.
This means that the execution will return to the CPU immediately after the kernel is launched.
Later we will see how this can be used to our advantage, since it allows us to keep CPU busy while GPU is executing the kernel.
But for the following example we will need to explicitly ask the CPU to wait until the GPU is done with the kernel execution.
This can be done with the following function from CUDA API:

cudaDeviceSynchronize(..)

__host__ __device__ cudaError_t cudaDeviceSynchronize()

We are already familiar with __host__ and __device__ specifiers: this function can be used in both host and device code.
As usual, the return type is cudaError_t, which may indicate that there was an error in execution and the function does not take any arguments.

This is all we are going to need for our next example, in which we are going to ask a thread to print its global index.

Exercise

Printing messages from the CUDA kernel

#include <stdio.h>

int main()
{
    printf("I am a CPU running one thread.\n");
    return 0;
}

#include <stdio.h>

__global__ void hello_kernel()
{
    printf("Hello from GPU thread %d\n", threadIdx.x);
}

int main()
{
    hello_kernel<<<1, 32>>>();

    cudaDeviceSynchronize();

    return 0;
}

#include <stdio.h>

__global__ void hello_kernel()
{
    int threadInBlock = threadIdx.x;
    int blockIndex = blockIdx.x;
    int blockSize = blockDim.x;
    int threadIndex = blockIndex * blockSize + threadInBlock;
    printf("Hello from GPU thread %d = %d * %d + %d\n", threadIndex, blockIndex, blockSize, threadInBlock);
}

int main()
{
    int numThreadsInBlock = 32;
    int numBlocks = 3;

    hello_kernel<<<numBlocks, numThreadsInBlock>>>();

    cudaDeviceSynchronize();

    return 0;
}

#include <stdio.h>

__global__ void hello_kernel()
{
    printf("Hello from GPU thread (%d, %d) = (%d, %d) * (%d, %d) + (%d, %d)\n",
        threadIdx.x, threadIdx.y,
        blockIdx.x, blockIdx.y,
        blockDim.x, blockDim.y,
        blockIdx.x * blockDim.x + threadIdx.x, blockIdx.y * blockDim.y + threadIdx.y);
}

int main()
{
    dim3 numThreadsInBlock(8, 4);
    dim3 numBlocks(1, 2);

    hello_kernel<<<numBlocks, numThreadsInBlock>>>();

    cudaDeviceSynchronize();

    return 0;
}

Change the file extension to .cu to inform the compiler that it will contain GPU code.

Create a kernel function. Remember that kernel should be marked with __global__ specifier and should return void.

In the kernel function, get the thread index using threadIdx.x and print it out.

Call the kernel in a single block of 32 threads.

Add cudaDeviceSynchronize(..) call after the kernel call to ensure that the host will wait for the GPU to complete the task.

What will happen if we don’t add the cudaDeviceSynchronize(..) call?

Everything will execute as normal, the CPU will wait for the GPU to complete the execution before terminating.

An error will occur since the GPU will not be able to complete the task before the end of the program is reached.

Only some of the threads will print their indices.

Nothing will be printed.

Solution

The correct answer is 4: nothing will be printed since the program termination is right after the kernel launch.
You can also add a sleep(..) function call after the kernel to ensure that it completes before the program terminates (make sure to include unistd.h to make the sleep function available).

Compile the code using nvcc, run the executable.

Modify the code to run in 4 blocks of 32 threads.
Apart from threadIdx.x, wou will need blockIdx.x and blockDim.x to compute the “global” thread index.
Print these values and the computed global index.

(*) Modify the code to use two-dimensional grid.
Remember, that the total number of threads per block is limited by 1024 on NVIDIA GPUs.

Why the order of the threads in the output is random?

Try executing the program several times to see if there is a pattern in the way the output is printed.
Try increasing the number of threads per block to 64.
Can you notice anything interesting in the order of threads within the block?

Solution

Driver assigns the threads to multiprocessors by blocks.
There is no guarantee that the first multiprocessor will complete its operations before the second.
The output is printed as the threads execution reach the corresponding line of the code and which one will be there faster depends on many different factors.
Within the block, the order seems to be consistent if the block size is 32.
When the number is larger, you can notice that the order of threads within chunks of 32 threads is consistent.
However, the order of this chunks can vary.
On NVIDIA GPU, execution is performed by so-called warps of threads and the size of a warp is exactly 32 for all NVIDIA GPUs.
Within the warp, the threads execute the same command simultaneously.
This is why the order within warp is consistent.
And this is also why one has to be very careful with thread divergency within warp.
Even if just one thread diverges within warp, the rest of the threads will wait until the divergent thread completes its operations.

© Copyright 2021, Artem Zhmurov and individual contributors..


Lecture: Manycore GPU Architectures
and Programming, Part 2
1
CSCE 569 Parallel Computing
Department of Computer Science and Engineering
Yonghong Yan
yanyh@cse.sc.edu
https://passlab.github.io/CSCE569/
Manycore GPU Architectures and
Programming: Outline
• Introduction
– GPU architectures, GPGPUs, and CUDA
• GPU Execution model
• CUDA Programming model
• Working with Memory in CUDA
– Global memory, shared and constant memory
• Streams and concurrency
• CUDA instruction intrinsic and library
• Performance, profiling, debugging, and error handling
• Directive-based high-level programming model
– OpenACC and OpenMP
2
How is the GPU controlled?
• The CUDA API is split into:
– The CUDA Management API
– The CUDA Kernel API
• The CUDA Management API is for a variety of operations
– GPU memory allocation, data transfer, execution, resource
creation
– Mostly regular C function and calls
• The CUDA Kernel API is used to define the computation to
be performed by the GPU
– C extensions
3
How is the GPU controlled?
• A CUDA kernel:
– Defines the operations to be performed by a single thread on
the GPU
– Just as a C/C++ function defines work to be done on the CPU
– Syntactically, a kernel looks like C/C++ with some extensions
__global__ void kernel(...) {
...
}
– Every CUDA thread executes the same kernel logic (SIMT)
– Initially, the only difference between threads are their thread
coordinates
4
How are GPU threads organized?
• CUDA thread hierarchy
– Warp = SIMT Group
– Thread Block = SIMT Groups that run
concurrently on an SM
– Grid = All Thread Blocks created by the same
kernel launch
• Launching a kernel is simple and similar to a function call.
– kernel name and arguments
– # of thread blocks/grid and # of threads/block to create:
kernel<<<nblocks,
threads_per_block>>>(arg1, arg2, ...);
5
How are GPU threads organized?
• In CUDA, only thread blocks and grids are first-class
citizens of the programming model.
• The number of warps created and their organization are
implicitly controlled by the kernel launch configuration, but
never set explicitly.
kernel<<<nblocks,
threads_per_block>>>(arg1, arg2, ...);
kernel launch
configuration
6
How are GPU threads organized?
• GPU threads can be configured in one-, two-, or three-
dimensional layouts
– One-dimensional blocks and grids:
int nblocks = 4;
int threads_per_block = 8;
kernel<<<nblocks, threads_per_block>>>(...);
7
Block 0
Block 1
Block 2
Block 3
How are GPU threads organized?
• GPU threads can be configured in one-, two-, or three-
dimensional layouts
– Two-dimensional blocks and grids:
dim3 nblocks(2,2)
dim3 threads_per_block(4,2);
kernel<<<nblocks, threads_per_block>>>(...);
8
How are GPU threads organized?
• GPU threads can be configured in one-, two-, or three-
dimensional layouts
– Two-dimensional grid and one-dimensional blocks:
dim3 nblocks(2,2);
int threads_per_block = 8;
kernel<<<nblocks, threads_per_block>>>(...);
9
How are GPU threads organized?
• On the GPU, the number of blocks and threads per block is
exposed through intrinsic thread coordinate variables:
– Dimensions
– IDs
Variable
Meaning
gridDim.x, gridDim.y,
gridDim.z
Number of blocks in a kernel
launch.
blockIdx.x, blockIdx.y,
blockIdx.z
Unique ID of the block that
contains the current thread.
blockDim.x, blockDim.y,
blockDim.z
Number of threads in each block.
threadIdx.x, threadIdx.y,
threadIdx.z
Unique ID of the current thread
within its block.
10
How are GPU threads organized?
to calculate a globally unique ID for a thread on the GPU
inside a one-dimensional grid and one-dimensional block:
kernel<<<4, 8>>>(...);
__global__ void kernel(...) {
int tid = blockIdx.x * blockDim.x + threadIdx.x;
...
}
11
Block 0
Block 1
Block 2
Block 3
blockIdx.x = 2;
blockDim.x = 8;
threadIdx.x = 2;
0  1  2  3  4  5  6  7
8
How are GPU threads organized?
• Thread coordinates offer a way to differentiate threads and
identify thread-specific input data or code paths.
– Link data and computation, a mapping
__global__ void kernel(int *arr) {
int tid = blockIdx.x * blockDim.x + threadIdx.x;
if (tid < 32) {
arr[tid] = f(arr[tid]);
} else {
arr[tid] = g(arr[tid]);
}
12
code path for threads with tid < 32
code path for threads with tid >= 32
Thread Divergence: recall that useless code path is
executed, but then disabled in SIMT execution model
How is GPU memory managed?
• CUDA Memory Management API
– Allocation of GPU memory
– Transfer of data from the host to GPU memory
– Free-ing GPU memory
– Foo(int A[][N]) { }
Host Function
CUDA Analogue
malloc
cudaMalloc
memcpy
cudaMemcpy
free
cudaFree
13
How is GPU memory managed?
cudaError_t cudaMalloc(void **devPtr,
size_t size);
– Allocate size bytes of GPU memory and store their address
at *devPtr
cudaError_t cudaFree(void *devPtr);
– Release the device memory allocation stored at devPtr
– Must be an allocation that was created using cudaMalloc
14
How is GPU memory managed?
cudaError_t cudaMemcpy(
void *dst, const void *src, size_t count,
enum cudaMemcpyKind kind);
– Transfers count bytes from the memory pointed to by src to
dst
– kind can be:
• cudaMemcpyHostToHost,
• cudaMemcpyHostToDevice,
• cudaMemcpyDeviceToHost,
• cudaMemcpyDeviceToDevice
– The locations of dst and src must match kind, e.g. if kind is
cudaMemcpyHostToDevice then src must be a host array and
dst must be a device array
15
How is GPU memory managed?
void *d_arr, *h_arr;
h_addr = … ; /* init host memory and data */
// Allocate memory on GPU and its address is in d_arr
cudaMalloc((void **)&d_arr, nbytes);
// Transfer data from host to device
cudaMemcpy(d_arr, h_arr, nbytes,
cudaMemcpyHostToDevice);
// Transfer data from a device to a host
cudaMemcpy(h_arr, d_arr, nbytes,
cudaMemcpyDeviceToHost);
// Free the allocated memory
cudaFree(d_arr);
16
CUDA Program Flow
• At its most basic, the flow of a CUDA program is as follows:
1. Allocate GPU memory
2. Populate GPU memory with inputs from the host
3. Execute a GPU kernel on those inputs
4. Transfer outputs from the GPU back to the host
5. Free GPU memory
• Let’s take a look at a simple example that manipulates data
17
AXPY Example with OpenMP: Multicore
• y = α·x + y
– x and y are vectors of size n
– α is scalar
• Data (x, y and a) are shared
– Parallelization is relatively easy
18
CUDA Program Flow
• AXPY is an embarrassingly parallel problem
– How can vector addition be parallelized?
– How can we map this to GPUs?
• Each thread does one element
A
B
C
19
AXPY Offloading To a GPU using CUDA
20
Memory allocation on device
Memcpy from host to device
Launch parallel execution
Memcpy from device to host
Deallocation of dev memory
CUDA Program Flow
• Consider the workflow of the example vector addition vecAdd.cu:
1.
Allocate space for A, B, and C on the GPU
2.
Transfer the initial contents of A and B to the GPU
3.
Execute a kernel in which each thread sums Ai and Bi, and stores the
result in Ci
4.
Transfer the final contents of C back to the host
5.
Free A, B, and C on the GPU
Modify to C = A+B+C
A = B*C;
we will need both C and A in the host side after GPU
computation.
• Compile and running on bridges:
– https://passlab.github.io/CSCE569/resources/HardwareSoftware.html#inte
ractive
– copy gpu_code_examples folder from my home folder
• cp –r ~yan/gpu_code_examples ~
– $nvcc –Xcompiler –fopenmp vectorAdd.cu
– $./a.out
21
More Examples and Exercises
• Matvec:
– Version 1: each thread computes one element of the final
vector
– Version 2:
• Matmul in assignment #4
– Version 1: each thread computes one row of the final matrix C
22
CUDA SDK Examples
• CUDA Programming Manual:
– http://docs.nvidia.com/cuda/cuda-c-programming-guide
• CUDA SDK Examples on bridges
– module load gcc/5.3.0 cuda/8.0
– export CUDA_PATH=/opt/packages/cuda/8.0
– /opt/packages/cuda/8.0/samples
• Copy to your home folder
– cp –r /opt/packages/cuda/8.0/samples ~/CUDA_samples
• Do a “make” in the folder, and it will build all the sources
• Or go to a specific example folder and make, it will build only the
binary
• Find ones you are interested in and run to see
23
Inspecting CUDA Programs
• Debugging CUDA program:
– cuda-gdb debugging tool, like gdb
• Profiling a program to examine the performance
– Nvprof tool, like gprof
– Nvprof ./vecAdd
24
Manycore GPU Architectures and
Programming: Outline
• Introduction
– GPU architectures, GPGPUs, and CUDA
• GPU Execution model
• CUDA Programming model
• Working with Memory in CUDA
– Global memory, shared and constant memory
• Streams and concurrency
• CUDA instruction intrinsic and library
• Performance, profiling, debugging, and error handling
• Directive-based high-level programming model
– OpenACC and OpenMP
25
Storing Data on the CPU
• A memory hierarchy emulates a large amount of low-
latency memory
– Cache data from a large, high-latency memory bank in a small
low-latency memory bank
DRAM
L2 Cache
L1 Cache
26
CPU Memory Hierarchy
CPU
GPU Memory Hierarchy
27
SIMT Thread Groups on a GPU
SIMT Thread Groups on an SM
SIMT Thread Group
Registers
Local Memory
On-Chip Shared
Memory/Cache
Global Memory
Constant Memory
Texture Memory
• More complex than
the CPU memory
– Many different types
of memory, each with
special-purpose
characteristics
• SRAM
• DRAM
– More explicit control
over data movement
Storing Data on the GPU
28
• Registers (SRAM)
– Lowest latency memory space on the
GPU
– Private to each CUDA thread
– Constant pool of registers per-SM
divided among threads in resident
thread blocks
– Architecture-dependent limit on
number of registers per thread
– Registers are not explicitly used by the
programmer, implicitly allocated by
the compiler
– -maxrregcount compiler option
allows you to limit # registers per
thread
Storing Data on the GPU
Shared Memory
Transfer
29
• Shared Memory (SRAM)
– Declared with the __shared__
keyword
– Low-latency, high bandwidth
– Shared by all threads in a thread block
– Explicitly allocated and managed by
the programmer, manual L1 cache
– Stored on-SM, same physical memory
as the GPU L1 cache
– On-SM memory is statically
partitioned between L1 cache and
shared memory
Storing Data on the GPU
L1 Cache
L2 Cache
Global Memory
30
• GPU Caches (SRAM)
– Behaviour of GPU caches is
architecture-dependent
– Per-SM L1 cache stored on-chip
– Per-GPU L2 cache stored off-chip,
caches values for all SMs
– Due to parallelism of accesses, GPU
caches do not follow the same LRU
rules as CPU caches
Storing Data on the GPU
Constant
Memory
Constant Cache
31
• Constant Memory (DRAM)
– Declared with the __constant__
keyword
– Read-only
– Limited in size: 64KB
– Stored in device memory (same
physical location as Global Memory)
– Cached in a per-SM constant cache
– Optimized for all threads in a warp
accessing the same memory cell
Storing Data on the GPU
Texture Memory
Read-Only Cache
Texture Memory
32
• Texture Memory (DRAM)
– Read-only
– Stored in device memory (same
physical location as Global Memory)
– Cached in a texture-only on-SM cache
– Optimized for 2D spatial locality
(caches commonly only optimized for
1D locality)
– Explicitly used by the programmer
– Special-purpose memory
Storing Data on the GPU
L1 Cache
L2 Cache
Global Memory
33
• Global Memory (DRAM)
– Large, high-latency memory
– Stored in device memory (along
with constant and texture memory)
– Can be declared statically with
__device__
– Can be allocated dynamically with
cudaMalloc
– Explicitly managed by the
programmer
– Optimized for all threads in a warp
accessing neighbouring memory
cells
Storing Data on the GPU
34
Storing Data on the GPU
35
Static Global Memory
• Static Global Memory has a fixed size throughout execution
time:
__device__ float devData;
__global__ void checkGlobalVariable()
printf(“devData has value %f\n”, devData);
}
• Initialized using cudaMemcpyToSymbol:
cudaMemcpyToSymbol(devData, &hostData, sizeof(float));
• Fetched using cudaMemcpyFromSymbol:
cudaMemcpyFromSymbol(&hostData, devData,
sizeof(float));
36
Dynamic Global Memory
• We have already seen dynamic global memory
– cudaMalloc dynamically allocates global memory
– cudaMemcpy transfers to/from global memory
– cudaFree frees global memory allocated by cudaMalloc
• cudaMemcpy supports 4 types of transfer:
– cudaMemcpyHostToHost,
cudaMemcpyHostToDevice,
cudaMemcpyDeviceToHost,
cudaMemcpyDeviceToDevice
• You can also memset global memory
cudaError_t cudaMemset(void *devPtr, int value,
size_t count);
37
Global Memory Access Patterns
• CPU caches are optimized for linear, iterative memory
accesses
– Cache lines ensure that accessing one memory cell brings
neighbouring memory cells into cache
– If an application exhibits good spatial or temporal locality
(which many do), later references will also hit in cache
CPU
System
Memory
Cache
38
Global Memory Access Patterns
• GPU caching is a more challenging problem
– Thousands of threads cooperating on a problem
– Difficult to predict the next round of accesses for all threads
• For efficient global memory access, GPUs instead rely on:
– Large device memory bandwidth
– Aligned and coalesced memory access patterns
– Maintaining sufficient pending I/O operations to keep the
memory bus saturated and hide global memory latency
39
Global Memory Access Patterns
• Achieving aligned and coalesced global memory accesses is
key to optimizing an application’s use of global memory
bandwidth
– Coalesced: the threads within a warp reference memory
addresses that can all be serviced by a single global memory
transaction (think of a memory transaction as the process of
bring a cache line into the cache)
– Aligned: the global memory accesses by threads within a warp
start at an address boundary that is an even multiple of the
size of a global memory transaction
40
Global Memory Access Patterns
• A global memory transaction is either 32 or 128 bytes
– The size of a memory transaction depends on which caches it
passes through
– If L1 + L2: 128 byte
– If L2 only: 32 bytes
– Which caches a global memory transaction passes through
depends on GPU architecture and the type of access (read vs.
write)
41
Global Memory Access Patterns
• Aligned and Coalesced Memory Access (w/ L1 cache)
– 32-thread wrap, 128-bytes memory transaction
• With 128-byte access, a single transaction is required and
all of the loaded bytes are used
42
Global Memory Access Patterns
• Misaligned and Coalesced Memory Access (w/ L1 cache)
• With 128-byte access, two memory transactions are
required to load all requested bytes. Only half of the loaded
bytes are used.
43
Global Memory Access Patterns
• Misaligned and Uncoalesced Memory Access (w/ L1 cache)
• With uncoalesced loads, many more bytes loaded than
requested
44
Global Memory Access Patterns
• Misaligned and Uncoalesced Memory Access (w/ L1 cache)
• One factor to consider with uncoalesced loads: while the
efficiency of this access is very low it may bring many cache
lines into L1/L2 cache which are used by later memory
accesses. The GPU is flexible enough to perform well, even
for applications that present suboptimal access patterns.
45
Global Memory Access Patterns
• Memory accesses that are not cached in L1 cache are
serviced by 32-byte transactions
– This can improve memory bandwidth utilization
– However, the L2 cache is device-wide, higher latency than L1,
and still relatively small è many applications may take a
performance hit if L1 cache is not used for reads
46
Global Memory Access Patterns
• Aligned and Coalesced Memory Access (w/o L1 cache)
• With 32-byte transactions, four transactions are required
and all of the loaded bytes are used
47
Global Memory Access Patterns
• Misaligned and Coalesced Memory Access (w/o L1 cache)
• With 32-byte transactions, extra memory transactions are
still required to load all requested bytes but the number of
wasted bytes is likely reduced, compared to 128-byte
transactions.
48
Global Memory Access Patterns
• Misaligned and Uncoalesced Memory Access (w/o L1
cache)
• With uncoalesced loads, more bytes loaded than requested
but better efficiency than with 128-byte transactions
49
Global Memory Access Patterns
• Global Memory Writes are always serviced by 32-byte
transactions
50
Global Memory and Special-Purpose Memory
• Global memory is widely useful and as easy to use as CPU DRAM
• Limitations
– Easy to find applications with memory access patterns that are
intrinsically poor for global memory
– Many threads accessing the same memory cell è poor global
memory efficiency
– Many threads accessing sparse memory cells è poor global
memory efficiency
• Special-purpose memory spaces to address these deficiencies in
global memory
– Specialized for different types of data, different access patterns
– Give more control over data movement and data placement than
CPU architectures do
53
Shared Memory
Shared Memory
Transfer
54
• Declared with the __shared__
keyword
• Low-latency, high bandwidth
• Shared by all threads in a thread
block
• Explicitly allocated and managed by
the programmer, manual L1 cache
• Stored on-SM, same physical memory
as the GPU L1 cache
• On-SM memory is statically
partitioned between L1 cache and
shared memory
Shared Memory Allocation
• Shared memory can be allocated statically or dynamically
• Statically Allocated Shared Memory
– Size is fixed at compile-time
– Can declare many statically allocated shared memory variables
– Can be declared globally or inside a device function
– Can be multi-dimensional arrays
__shared__ int s_arr[256][256];
55
Shared Memory Allocation
• Dynamically Allocated Shared Memory
– Size in bytes is set at kernel launch with a third kernel launch
configurable
– Can only have one dynamically allocated shared memory array
per kernel
– Must be one-dimensional arrays
__global__ void kernel(...) {
extern __shared__ int s_arr[];
...
}
kernel<<<nblocks, threads_per_block,
shared_memory_bytes>>>(...);
56
Matvec using shared memory
57
Matrix Vector Multiplication
58
Matrix Vector Multiplication
59
Matrix Multiplication V1 and V2 in Assignment
#4
• https://docs.nvidia.com/cuda/cuda-c-programming-
guide/#shared-memory
60
GPU Memory Hierarchy
68
SIMT Thread Groups on a GPU
SIMT Thread Groups on an SM
SIMT Thread Group
Registers
Local Memory
On-Chip Shared
Memory/Cache
Global Memory
Constant Memory
Texture Memory
• More complex than
the CPU memory
– Many different types
of memory, each with
special-purpose
characteristics
• SRAM
• DRAM
– More explicit control
over data movement
Constant Memory
Constant
Memory
Constant Cache
69
• Declared with the __constant__
keyword
• Read-only
• Limited in size: 64KB
• Stored in device memory (same
physical location as Global Memory)
• Cached in a per-SM constant cache
• Optimized for all threads in a warp
accessing the same memory cell
Constant Memory
• As its name suggests, constant memory is best used for
storing constants
– Values which are read-only
– Values that are accessed identically by all threads
• For example: suppose all threads are evaluating the
equation
y = mx + b
for different values of x, but identical values of m and b
– All threads would reference m and b with the same memory
operation
– This broadcast access pattern is optimal for constant memory
70
Constant Memory
• A simple 1D stencil
– target cell is updated based on its 8 neighbors, weighted by
some constants c0, c1, c2, c3
71
Constant Memory
• constantStencil.cu contains an example 1D stencil
that uses constant memory
__constant__ float coef[RADIUS + 1];
cudaMemcpyToSymbol(coef, h_coef, (RADIUS + 1) *
sizeof(float));
__global__ void stencil_1d(float *in, float *out, int N)
{
...
for (int i = 1; i <= RADIUS; i++) {
tmp += coef[i] * (smem[sidx + i] - smem[sidx - i]);
}
}
72
CUDA Synchronization
• When using shared memory, you often must coordinate
accesses by multiple threads to the same data
• CUDA offers synchronization primitives that allow you to
synchronize among threads
73
CUDA Synchronization
__syncthreads
– Synchronizes execution across all threads in a thread block
– No thread in a thread block can progress past a __syncthreads
before all other threads have reached it
– __syncthreads ensures that all changes to shared and
global memory by threads in this block are visible to all other
threads in this block
__threadfence_block
– All writes to shared and global memory by the calling thread
are visible to all other threads in its block after this fence
– Does not block thread execution
74
CUDA Synchronization
__threadfence
– All writes to global memory by the calling thread are visible to
all other threads in its grid after this fence
– Does not block thread execution
__threadfence_system
– All writes to global memory, page-locked host memory, and
memory of other CUDA devices by the calling thread are
visible to all other threads on all CUDA devices and all host
threads after this fence
– Does not block thread execution
75
Suggested Readings
1. Chapter 2, 4, 5 in Professional CUDA C Programming
2. Cliff Woolley. GPU Optimization Fundamentals. 2013.
https://www.olcf.ornl.gov/ wp-
content/uploads/2013/02/GPU_Opt_Fund-CW1.pdf
3. Mark Harris. Using Shared Memory in CUDA C/C++.
http://devblogs.nvidia.com/ parallelforall/using-shared-
memory-cuda-cc/
4. Mark Harris. Optimizing Parallel Reduction in CUDA.
http://developer.download.nvidia
.com/assets/cuda/files/reduction.pdf
76


CUDA/HIP

GPU Accelerated Small Matrix Multiplications

libsmm_acc is a library for small matrix-matrix multiplication on a GPU-accelerator. Stacks of matrix-matrix multiplication indices are passed from DBCSR to libsmm_acc which performs the multiplications on the GPU.

For a description of the library (some details are outdated, but this nevertheless provides a very good introduction), see Chapter 8.4 of:

WALKER, R. C., & GOETZ, A. W. (2016). Electronic structure calculations on graphics processing units: from quantum chemistry to condensed matter physics.

Compilation

libsmm_acc is compiled from within DBCSR, there is no separate compilation.

Directory Organization

Matrix-matrix Multiplication Kernels and Parameters

For a given matrix-matrix multiplication triplet characterized by dimensions

libsmm_acc can run 5 different matrix-matrix multiplication kernels:

which take between 3 - 7 parameters (see figure at the top):

The performance of the matrix-matrix multiplication kernels is highly dependent on the choice of algorithm and parameters. For this reason, libsmm_acc provides lists of optimal parameters for different GPU cards and different (m, n, k)-triplets.

Contributing to libsmm_acc

We expect users to contribute to the library by providing new optimized kernels and support for new GPUs.

Autotuning procedure

Follow the autotuning procedure

Adding a new kernel

Choose a kernel name

Add the kernel's code (must be able to compile by both nvcc and hip) in file kernels/smm_acc_dnt_name.h

Add Python kernel class inheriting from base class kernels/smm_acc_dnt_name.py

Adding support for a new GPU card

Add the GPU's compute architecture properties to kernels/gpu_properties.json. For more information on where to find these properties, please refer to the "info" field of kernels/gpu_properties.json.

Add the GPU to the gpu_architectures data structure in kernels/smm_acc.py.

Add the necessary code for setting ARCH_NUMBER correctly in the CMakeLists. Also add this GPU to the list of SUPPORTED_CUDA_ARCHITECTURES or SUPPORTED_HIP_ARCHITECTURES in the CMakeLists.

Add a minimal JSON file parameters_GPU.json, containing:

{
}

then add matrix-matrix multiplication parameters for this GPU using autotuning.

DBCSR
 was developed by DBCSR Authors              © 2025

Documentation generated by
              FORD