How to optimize my cuda code?

I have a simple program, I just want to verify my GPU real performance. but, its result is out of my expectation. I don’t know how to explain it, and how to optimize my program. So, I hope NV’s experts can help me.

the detail about my GPU as following:
Device 0: “NVIDIA RTX A4000”
CUDA Driver Version / Runtime Version          11.6 / 11.3
CUDA Capability Major/Minor version number:    8.6
Total amount of global memory:                 16109 MBytes (16891379712 bytes)
(48) Multiprocessors, (128) CUDA Cores/MP:     6144 CUDA Cores
GPU Max Clock rate:                            1560 MHz (1.56 GHz)
Memory Clock rate:                             7001 Mhz
Memory Bus Width:                              256-bit
L2 Cache Size:                                 4194304 bytes
Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
Total amount of constant memory:               65536 bytes
Total amount of shared memory per block:       49152 bytes
Total shared memory per multiprocessor:        102400 bytes
Total number of registers available per block: 65536
Warp size:                                     32
Maximum number of threads per multiprocessor:  1536
Maximum number of threads per block:           1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch:                          2147483647 bytes
Texture alignment:                             512 bytes
Concurrent copy and kernel execution:          Yes with 2 copy engine(s)

ok, let me introduce my program, my program is very very simple, just execut “ffma”, details as following:

the performance should be ~10Tflops(x2=20Tflops)，but the result of my program is about 6.5T, just about 65% peak performance.
I modified the blocksPerSM to 2, 4, 8, and I modified the threadPerBlock to 128/256/512。 unfortunately, these results are very similar, about 60% - 65%。

and then, I profiled my program with NCU，it tell me “The ratio of peak float (fp32) to double (fp64) performance on this device is 64:1. The kernel achieved 61% of this device’s fp32 peak performance and 0% of its fp64 peak performance.” in “Roofline Analysis”

I think my program have avoid memory-access, avoid register dependency, I don’t know why my peak performance is 61%

I tried to read the profile information in NCU，but, I found I still cannot found the reason of poor performance.

I’ve uploaded my program and the profile file from NCU，
base_mac.tar.gz (3.7 KB)
repoprt.ncu-rep (12.6 MB)

Is there anyone would like to teach me? I think the keypoint is reading the profile file, but I cannot understand them, is there anyone would like to help me?

I implement the loop body with ptx, I think it can avoid the optimization behavior of nvcc.

Highly unlikely to be a good idea. The CUDA compiler is based on LLVM, an extremly powerful framework for code transformations, i.e. optimizations. If you run into the compiler optimizing away code that you don’t want to have optimized away, create dependencies that prevent that from happening. Your chosen approach for measuring peak FP32 throughput appears to be the common method of using independent dot products. You would want to sum these dot products at the end and write the result to global memory to avoid dead code elimination.

Instruction caches on GPUs tend to be pretty small, and your massive loop may exceed the size of the instruction cache, which based on past experience may cost 3% of performance.

It is easier to fill up the SMs as much as possible using relatively fine granularity, e.g. use 128 threads per thread block as a starting point.

You may be better off using dot-products floats instead of float4s. The latter is likely to result in higher register pressure.

You are unlikely to achieve more than 85% of theoretical FP32 throughput, as the ptxas compiler is unlikely to produce a perfect instruction scheduling with perfect register assignment. So there will be bubbles in the pipeline caused by register bank conflicts and execution pipe contention.

[Later: ]

Below is a simple test scaffold for measuring FP32 throughput. It is currently configured for the 9 year old low-end GPU in my web-browsing machine, for which it achieves 86% of theoretical peak FP32 throughput. Increase MAX_BLOCKS, REPS, ITER to adapt to your hardware. Then vary POLY_DEPTH to see how throughput changes.

#include <stdlib.h>
#include <stdio.h>

#define MAX_BLOCKS      (65520)
#define THREADS_PER_BLK (128)
#define LEN             (MAX_BLOCKS * 1024)
#define POLY_DEPTH      (512)
#define REPS            (2)
#define ITER            (10)

#if defined(_WIN32)
#if !defined(WIN32_LEAN_AND_MEAN)
#define WIN32_LEAN_AND_MEAN
#endif
#include <windows.h>
double second (void)
{
    LARGE_INTEGER t;
    static double oofreq;
    static int checkedForHighResTimer;
    static BOOL hasHighResTimer;

    if (!checkedForHighResTimer) {
        hasHighResTimer = QueryPerformanceFrequency (&t);
        oofreq = 1.0 / (double)t.QuadPart;
        checkedForHighResTimer = 1;
    }
    if (hasHighResTimer) {
        QueryPerformanceCounter (&t);
        return (double)t.QuadPart * oofreq;
    } else {
        return (double)GetTickCount() * 1.0e-3;
    }
}
#elif defined(__linux__) || defined(__APPLE__)
#include <stddef.h>
#include <sys/time.h>
double second (void)
{
    struct timeval tv;
    gettimeofday(&tv, NULL);
    return (double)tv.tv_sec + (double)tv.tv_usec * 1.0e-6;
}
#else
#error unsupported platform
#endif

// Macro to catch CUDA errors in CUDA runtime calls
#define CUDA_SAFE_CALL(call)                                          \
do {                                                                  \
    cudaError_t err = call;                                           \
    if (cudaSuccess != err) {                                         \
        fprintf (stderr, "Cuda error in file '%s' in line %i : %s.\n",\
                 __FILE__, __LINE__, cudaGetErrorString(err) );       \
        exit(EXIT_FAILURE);                                           \
    }                                                                 \
} while (0)

// Macro to catch CUDA errors in kernel launches
#define CHECK_LAUNCH_ERROR()                                          \
do {                                                                  \
    /* Check synchronous errors, i.e. pre-launch */                   \
    cudaError_t err = cudaGetLastError();                             \
    if (cudaSuccess != err) {                                         \
        fprintf (stderr, "Cuda error in file '%s' in line %i : %s.\n",\
                 __FILE__, __LINE__, cudaGetErrorString(err) );       \
        exit(EXIT_FAILURE);                                           \
    }                                                                 \
    /* Check asynchronous errors, i.e. kernel failed (ULF) */         \
    err = cudaDeviceSynchronize();                                    \
    if (cudaSuccess != err) {                                         \
        fprintf (stderr, "Cuda error in file '%s' in line %i : %s.\n",\
                 __FILE__, __LINE__, cudaGetErrorString( err) );      \
        exit(EXIT_FAILURE);                                           \
    }                                                                 \
} while (0)

__global__ void kernel (const float * __restrict__ src,
                        float * __restrict__ dst, int len)
{
    int stride = gridDim.x * blockDim.x;
    int tid = blockDim.x * blockIdx.x + threadIdx.x;
    for (int i = tid; i < len; i += stride) {
        float p = src[i] + 1.000001f;
        float q = src[i] + 1.000002f;
        for (int k = 0; k < REPS; k++) {
#pragma unroll POLY_DEPTH
            for (int j = 0; j < POLY_DEPTH; j++) {
                p = fmaf (p, p, 1.000001f);
                q = fmaf (q, q, 1.000002f);
            }
        }
        dst[i] = p + q;
    }
}

int main (int argc, char *argv[])
{
    double start, stop, nbr_of_fma;
    float *d_a, *d_b;

    /* Allocate memory on device */
    CUDA_SAFE_CALL (cudaMalloc((void**)&d_a, sizeof(d_a[0]) * LEN));
    CUDA_SAFE_CALL (cudaMalloc((void**)&d_b, sizeof(d_b[0]) * LEN));

    /* Initialize device memory */
    CUDA_SAFE_CALL (cudaMemset(d_a, 0x00, sizeof(d_a[0]) * LEN)); // zero

    /* Compute execution configuration */
    dim3 dimBlock(THREADS_PER_BLK);
    int threadBlocks = (LEN + (dimBlock.x - 1)) / dimBlock.x;
    dim3 dimGrid(threadBlocks);

    printf ("burn: using %d threads per block, %d blocks, %f GB used\n",
            dimBlock.x, dimGrid.x, 2*1e-9*LEN*sizeof(d_a[0]));

    start = second();
    for (int k = 0; k < ITER; k++) {
        kernel<<<dimGrid,dimBlock>>>(d_a, d_b, LEN);
        CHECK_LAUNCH_ERROR();
    }
    stop = second();
    nbr_of_fma = (2.0 * POLY_DEPTH * REPS + 3.0) * LEN * ITER;
    printf ("flop=%13.6e  elapsed=%.5f sec  throughput=%.5f FP32 GFLOPS\n",
            nbr_of_fma * 2, stop-start, nbr_of_fma * 2 * 1e-9 / (stop - start));

    CUDA_SAFE_CALL (cudaFree(d_a));
    CUDA_SAFE_CALL (cudaFree(d_b));

    return EXIT_SUCCESS;
}

so kindly, thanks for your code.
I run your code, and I get the performance above 90%,  so cool
but, I found a difference between my poor code and your good code.
I found the clause about fma in your code like this:

p = fmaf (p, p, 1.000001f);
                q = fmaf (q, q, 1.000002f);

at the same time, my code like this:

c += a * b;

and then, I modified your code like this:

__global__ void kernel (const float * __restrict__ src,
                        float * __restrict__ dst, int len)
{
    int stride = gridDim.x * blockDim.x;
    int tid = blockDim.x * blockIdx.x + threadIdx.x;
    for (int i = tid; i < len; i += stride) {
        float p = src[i] + 1.000001f;
        float q = src[i] + 1.000002f;
        float r = 0.0f;
        for (int k = 0; k < REPS; k++) {
            #pragma unroll(512)
            for (int j = 0; j < POLY_DEPTH; j++) {
                r = fmaf (p, q, r);
            }
        }
        dst[i] = r * 0.0001;
    }
}

and modify the statistic clause like this:

nbr_of_fma = (POLY_DEPTH * REPS + 3.0) * LEN * ITER;

and then, the performance is about 50%.

as comparison, I modify my blockPerSM to 64, and modify the loop body like this:

#pragma unroll(128)
  for(int i = 0 ; i < loop ; i ++){
    vec_C.x = fmaf (vec_A.x, vec_B.x, vec_C.x);
    vec_C.y = fmaf (vec_A.y, vec_B.y, vec_C.y);
    vec_C.z = fmaf (vec_A.z, vec_B.z, vec_C.z);
    vec_C.w = fmaf (vec_A.w, vec_B.w, vec_C.w);
}
  *ptr_C = vec_C;

performance is about 60%.

another modifcation like this:

loop = loop >> 1;
  #pragma unroll(128)
  for(int i = 0 ; i < loop ; i ++){
    vec_A.x = fmaf (vec_A.x, vec_A.x, 0.0001f);
    vec_A.y = fmaf (vec_A.y, vec_A.y, 0.0001f);
    vec_A.z = fmaf (vec_A.z, vec_A.z, 0.0001f);
    vec_A.w = fmaf (vec_A.w, vec_A.w, 0.0001f);

    vec_B.x = fmaf (vec_B.x, vec_B.x, 0.0002f);
    vec_B.y = fmaf (vec_B.y, vec_B.y, 0.0002f);
    vec_B.z = fmaf (vec_B.z, vec_B.z, 0.0002f);
    vec_B.w = fmaf (vec_B.w, vec_B.w, 0.0002f);
  }
  vec_C.x = vec_A.x + vec_B.x;
  vec_C.y = vec_A.y + vec_B.y;
  vec_C.z = vec_A.z + vec_B.z;
  vec_C.w = vec_A.w + vec_B.w;

  *ptr_C = vec_C;

this performance is also about 90%,
I think the third parameter of fmaf is constant, so, I feel this result(90%) cannot prove the real performance.

I think the keypoint is “register reuse” and data dependency, but your code there are dependency(a = a * a + 1.0001f) also, and the performance is good. why?

I checkd their sass,
the sass about the fma in your code, like this:

/*0130*/                   FFMA R6, R6, R6, 1.0000009536743164062 ;       /* 0x3f80000806067423 */
                                                                                /* 0x000fe20000000006 */
        /*0140*/                   FFMA R7, R7, R7, 1.0000020265579223633 ;       /* 0x3f80001107077423 */
                                                                                /* 0x000fc60000000007 */
        /*0150*/                   FFMA R6, R6, R6, 1.0000009536743164062 ;       /* 0x3f80000806067423 */
                                                                                /* 0x000fe20000000006 */
        /*0160*/                   FFMA R7, R7, R7, 1.0000020265579223633 ;       /* 0x3f80001107077423 */
                                                                                /* 0x000fc60000000007 */

the sass about fma in my code, like this:

/*0190*/                   FFMA R17, R4, R8, R12 ;                      /* 0x0000000804117223 */
                                                                                /* 0x020fe2000000000c */
        /*01a0*/                   FFMA R12, R5, R9, R13 ;                      /* 0x00000009050c7223 */
                                                                                /* 0x000fe2000000000d */
        /*01b0*/                   FFMA R13, R6, R10, R14 ;                     /* 0x0000000a060d7223 */
                                                                                /* 0x000fe2000000000e */
        /*01c0*/                   FFMA R14, R7, R11, R15 ;                     /* 0x0000000b070e7223 */
                                                                                /* 0x000fe2000000000f */
        /*01d0*/                   FFMA R17, R4, R8, R17 ;                      /* 0x0000000804117223 */
                                                                                /* 0x000fe20000000011 */
        /*01e0*/                   FFMA R12, R5, R9, R12 ;                      /* 0x00000009050c7223 */
                                                                                /* 0x000fe2000000000c */
        /*01f0*/                   FFMA R13, R6, R10, R13 ;                     /* 0x0000000a060d7223 */
                                                                                /* 0x000fe2000000000d */
        /*0200*/                   FFMA R14, R7, R11, R14 ;                     /* 0x0000000b070e7223 */
                                                                                /* 0x000fe2000000000e */
        /*0210*/                   FFMA R17, R4, R8, R17 ;                      /* 0x0000000804117223 */
                                                                                /* 0x000fe20000000011 */
        /*0220*/                   FFMA R12, R5, R9, R12 ;                      /* 0x00000009050c7223 */
                                                                                /* 0x000fe2000000000c */
        /*0230*/                   FFMA R13, R6, R10, R13 ;                     /* 0x0000000a060d7223 */
                                                                                /* 0x000fe2000000000d */
        /*0240*/                   FFMA R14, R7, R11, R14 ;                     /* 0x0000000b070e7223 */
                                                                                /* 0x000fe2000000000e */

obviously:

So, would you like to teach me how to avoid these dependency, if these dependency really exist.

The code I posted above is an ad-hoc adaption of some code I have had sitting around for quite a few years. I seem to recall that I chose the particular arrangement of FMAs used so as to minimize register bank conflicts, but I do not know for sure.

Having been retired for almost a decade, I am a hobbyist these days who will, often on a whim, explore some issue for an extended afternoon, then forget the exploratory code once my curiosity is satisfied: “The journey is the reward”. I rarely keep notes on what I tried and why.

Sustaining three-input operations at full speed is a challenge in all processor architectures due to the tremendous bandwidth required (3 read ports, 1 write port on the register file). The problem is exacerbated by multi-issue capability. A common way to boost register file bandwidth on average is to use register banks, each of which provides fewer read (and possibly, write) ports. To my knowledge, all NVIDIA GPUs use a (publicly undocumented) scheme of this nature to boost practically available bandwidth. Bank conflicts causing pipeline bubbles may occur intra-instruction or inter-instruction in case of multi-issue capability.

thanks for your clear explanation
I see，R4, R8, R12 is conflict（register bank, 0 == 4%4，0 == 8%4, 0 == 12%4， they are in the same bank，0）, and R5/R9/R13，R6/R10/R14, R7/R11/R15, they are all conflict. so,  performance is poor as your explanation, right?

/*0190*/                   FFMA R17, R4, R8, R12 ;                      /* 0x0000000804117223 */
                                                                                /* 0x020fe2000000000c */
        /*01a0*/                   FFMA R12, R5, R9, R13 ;                      /* 0x00000009050c7223 */
                                                                                /* 0x000fe2000000000d */
        /*01b0*/                   FFMA R13, R6, R10, R14 ;                     /* 0x0000000a060d7223 */
                                                                                /* 0x000fe2000000000e */
        /*01c0*/                   FFMA R14, R7, R11, R15 ;

but I don’t know how to solve it, because the SASS is complied by nvcc.
maybe, I don’t know some tricky, would you like to give me more further advise?

but I don’t know how to solve it, because the SASS is complied by nvcc.

The nvcc compiler is aware of performance issues related to register usage.   While its certainly possible that this can be improved, its also possible that this is the best tradeoff of choices about register usage.

It should be evident in a large dependent sequence like this that register usage changes will have side effects.  Changes you make to address performance on one instruction may have a negative performance impact elsewhere in the dependent chain.

I don’t know of “tricks” to tell nvcc to reorganize its register usage.  You could try playing with very crude, coarse controls like -maxrregcount switch to the compiler.

The other options I know of are:

Based on the 2nd option, you could file a bug to request study by the compiler team, if you think you can do better than the compiler.  But you would need a well-documented example, showing a wholistic solution.  Even then, there are probably knowledge gaps that make this sort of approach difficult.

To quote Wikipedia:

The problem of optimal register allocation is NP-complete. As a consequence, compilers employ heuristic techniques to approximate its solution.

I think it is reasonable to assume that the CUDA compiler engineers in charge of ptxas are fully aware of the latest developments in the field and that heuristics that consider the general problem constraint by GPU-specific restrictions (such as calling conventions , register aggregation for 64-bit operations or vector loads/stores, dual issue, and register banks) are in place. It would also be reasonable to assume that after 15 years of development, this fundamental building block of a compiler is mature.

That does not mean there could not be room for improvement, just that the burden of demonstrating noticeably improved performance from superior register allocation for specific cases lies with a prospective bug filer.

I just heard of register bank is “%4”，but, I didn’t find it in nv’s public document.
So, I must confirm the restriction for register bank at first, would you like to give me some documents about it?
my compiler is nvcc 11.8, and the arch is sm86

I am not aware that the details of the GPU register file organization are disclosed in official CUDA documentation. I also cannot find any relevant details in papers from people who have explored GPU microarchitectures with targeted microbenchmarks. An older paper by NVIDIA subject matter experts,

Mark Gebhart, Stephen W. Keckler, Brucek Khailany, Ronny Krashinsky, William J. Dally, “Unifying Primary Cache, Scratch, and Register File Memories in a Throughput Processor”

gives some details, but I have no idea how these relate to newer architectures:

Each MRF bank is 16 bytes wide with 4 bytes allocated to the
same-named architectural register for threads in each of the 4 SIMT
lanes in the cluster. Each bank has a capacity of 8KB, providing
a total of 256KB of register file capacity per SM. Registers are
interleaved across the register file banks to minimize bank conflicts.
Instructions that access multiple values from the same bank incur a
cycle of delay for each access beyond the first. The operand buffering
between the MRF and the execution units represents interconnect
and pipeline storage for operands that may be fetched from the MRF
on different cycles. Stalls due to bank conflicts are rare and can be
minimized with compiler techniques

Note that the use of vector types, such as float4, tends to impose an additional burden on register allocation. That is why earlier in this thread, I suggested using scalar computation for the exercise at hand (maximize floating-point throughput), and did so in the sample code I posted.

thanks for your patient

In practical terms, I see the risk of becoming mired in microarchitectural details that have no bearing on 99% of real-life CUDA code out there. Yes, it is cool to figure out how to get close to the theoretical peak FLOPS, but the resulting code often has very little similarity with code people write to address actual use cases. Not much has changed in that regard since I showed how to get 1 GFLOPS out of the AMD K7 (Athlon) processor ca. 1999.

Outside of special scenarios, much forward progress can be made by simply relying on the CUDA compiler and using feedback provided by the CUDA profiler.

“relying on the CUDA compiler and using feedback provided by the CUDA profiler”， yes, you are right
but, my dilemma is I cannot obtain useful clue from NCU’s profile information.

You might want to start with the recommendations from the CUDA Best Practices guide before delving into the profiler. All profilers these days are very sophisticated utilities that one needs to spend some quality time with to get the full benefit. So give it some time, keep on experimenting and exploring.

When profiler use first became common some thirty years ago, their functionality was very limited and interacting with profilers was less overwhelming than today. There is a bit of a trade-off between ease of use and depth of analysis, I would say.

So, I must confirm the restriction for register bank at first, would you like to give me some documents about it?

One source of information is pages 6-8 of Dissecting Volta. The same authors wrote Dissecting Ampere, where they state on page 29, that the Ampere register layout is the same.

thanks for your help

Related topics

Powered by Discourse, best viewed with JavaScript enabled


An Almost Pointless Exercise in GPU Optimization

Not everyone is able to write funky
fused
operators to make ML models run faster on GPUs using clever quantisation tricks. However lots of
developers work with algorithms that feel like they should be able to leverage the thousands of
cores in a GPU to run faster than using the dozens of cores on a server CPU. To see what is possible
and what is involved, I revisited the first problem I ever considered trying to accelerate with a
GPU. What is unusual about my chosen problem is that it is officially
pointless, so you ought not to
be able to find any library that will accelerate this algorithm, because it isnât worth writing one!
That makes it an interesting proxy for algorithms which arenât catered for by high-performance
libraries written by experts, but can be structured to run thousands of threads in parallel.

TL;DRâ

Getting an existing C++ algorithm running on GPU is pretty easy, so it is a low bar to get started.
What I learned is the importance of minimizing thread divergence and maximizing effective memory
access speed. To do that effectively, I had to transform my algorithm into a state machine structure
so that every thread is operating mostly in lock-step, just with different data values.

My starting, interim and final code are open to see, along with a
summary of the steps I took, and the corresponding improvements or regressions at each stage. I want
to focus in this article on the thought process for deciding each step, mostly by explaining the
Nvidia Nsight Compute analysis which helped guide me.

In the end I managed to make my program run about 30x faster on my laptop using its GeForce GTX 1650
GPU, compared with its Core i7â9750H CPU. Only in the last two steps did it get meaningfully better
than with CPU though, so be prepared for early and frequent disappointment.

If you want just the summary of what worked, jump to Progression History.

A Pointless Programâ

Years ago, a colleague invited me to take on his Christmas programming challenge, which was to write
the fastest program he could to continuously play the card game Beggar My Neighbour. The aim, noted
by John Conway as definitely not worth
solving, is to try to find the
longest game â with the possibility that there might be a game which never ends. The longest game
found so far has 8344 turns â a rainy afternoon diversion perhaps, if you can sustain playing a card
every 2.589 seconds for six hours straight! You can see a history of new records, with a Python
program that verifies them, here:
https://github.com/matthewmayer/beggarmypython

The game play algorithm is almost trivial, but it has some notable features which turn out to be
relevant to the challenge of effectively leveraging a GPU.

Game play is completely deterministic, so the outcome is defined only by the initial sorting of
the deck. The problem is therefore embarrassingly parallel, since there are
653,534,134,886,878,245,000 (roughly 6.535Ã10206.535 \times 10^{20}6.535Ã1020) distinct deals, and we can play as
many as we want in parallel, in any order.

The algorithm tracks game state by using a few state variables and nested branching logic. This
is easy to write and verify, and actually reasonably efficient to run on a single CPU core.

The algorithm is very compact in terms of code size and data size. The search loop to play many
games while tracking the best ones so far is also very compact.

CPU Starting Pointâ

My initial C++ program to search for long games is
here. It is just a port of the Python
program with a search loop that runs continuously, shuffling the deck randomly between games. I
implemented two simple optimizations that seemed obvious:

a 64-entry circular buffer for each playerâs hand and the discard pile, combined with a bit mask
step to dereference the first and last index pointers. This avoids extra instructions and branches
to handle wrap-around of the first and last index values as cards are added and removed.

swap two cards between each game, to use fewer random numbers â we donât need each deal to be
completely random, but just have a random difference from the previous one.

The program can use multiple CPU cores by just running separate copies of the search loop in
different threads, starting with different RNG seeds. Each search loop does its own random walk
through potential deals.

On my laptop, throughput peaks at 2.9 million deals per second, using 12 threads to match the
number of logical CPU cores.

Initial GPU Portâ

The beauty of General Purpose GPU programming is that you can often get started quickly with the
code you already have. There is a good chance that only a few adaptations are needed to convert a
program that uses multiple threads on CPU to one that runs with even more threads on GPU, as long as
you donât have extreme memory requirements (for code or data or both) which force breaking up the
work into smaller pieces.

GPU cores have a similar range of machine instructions to those of CPUs, so plain algorithms will
compile readily enough to give reasonable single core efficiency; you donât need to transform your
program to use a subset of variable types or data structures or code constructs, or special
parallelization primitives or libraries. You mainly just need to cope with some changes to library
functions (such as the random number generator in my case), and finessing of class structures to
graft in the global functions needed to launch work on the GPU.

You do need to be ready for disappointment though.

If your code is similar to mine, founded on nested branching logic, you may find that the GPU wonât
even be able to match the CPU performance at first. This is to be expected: CPUs are designed for
running unrelated complex branching logic in each thread, dedicating significant chip area on
circuitry to predict branches, speculatively execute multiple paths, and do lots of caching â 1 MB
per logical CPU core in my laptop for instance. CPU cores also run fast, with 3â4GHz max speed being
fairly typical. By comparison, GPUs are much lighter on hardware per core for caching, branch
prediction etc, and top out at perhaps 1.5â2GHz (unless you have an extreme cooling gaming GPU).
They come with faster memory though, to help compensate.

But the net effect is that an algorithm will probably run 2â3 times faster on a single CPU core
than it will on a single GPU core, from a combination of the clock speed ratio and more aggressive
single thread acceleration. You need to figure out how to make good use of the thousands of slow GPU
cores in order to outperform a few fast CPU cores.

My initial port to GPU ran at about
1.4M deals per second (once thermal throttling kicks in), using 2048 threads (128 blocks of 16
threads each). Not an encouraging start, but in hindsight about what I should have expected.

Learn to use Nsight Computeâ

Early on, I decided to get comfortable using the Nsight Compute tool to analyse the GPU portion of
my code in spectacular detail, rather than trying to rely on intuition and the high-level
utilization figures from nvidia-smi. The approach I settled on was to step through the execution of
the first couple of GPU kernel launches (the kernel here being my global function which runs the
core search and game play loop for a predefined number of iterations), and then profile the next
kernel run to see how much of the hardware capability was being used effectively.

The first, unflattering, report that the tool shows is the âSpeed of Lightâ summary: ie. the
percentage of theoretical peak performance on compute and memory bandwidth utilization. My first
program scored around 12% for compute, and 28% for memory. (For the same program, nvidia-smi would
report 88% and 28% utilization respectively, underscoring how misleading its information can be when
you are trying to optimize your algorithm in the early stages.)

There were many detailed metrics underneath the headline figures which could be examined, and
various analysis warnings that point out potential problem areas. Many of these sounded esoteric,
but there were two reasonably clear actionable warnings:

The first one was easy to address: assign at least 32 threads per block (not 16), so we donât leave
warp capacity unused for not even trying. I ended up settling on 32 blocks of 32 threads, which
increased throughput to 2.3M deals per second. nvidia-smi now reported 95% and 13% for compute
and memory utilization. Nsight Compute reports we have improved average active threads per warp from
2.4 to 3.6. That left Thread Divergence as the key warning to address.

Thread Divergenceâ

If you have learned about CUDA programming before, you may recall that thread divergence occurs when
not all threads in the same warp are executing the same instruction. (Each group of 32 consecutive
threads in a block will always execute together as a unit, known as a warp, and GPU hardware is
heavily optimized around running these warps of threads very efficiently on the assumption that the
threads are normally executing instructions in unison, working on closely related pieces of the same
computation.)

It is part of the beauty of General Purpose GPU programming that thread divergence is allowed, and
is handled automatically by the hardware as well as it can be, by simply letting the threads in the
warp take turns to run their next instruction, when they are different. Subsets of the warp that do
share the next instruction will execute together, so degradation in performance is proportional to
how much the threads have diverged from each other.

In the worst case, where every thread is at its own unique point in the code, the threads are
effectively time sliced on the hardware, and so running at 1/32 of the hardwareâs potential. From
the Nsight Compute warning, I could see we were quite close to that point:

[Warning] Instructions are executed in warps, which are groups of 32 threads. Optimal instruction
throughput is achieved if all 32 threads of a warp execute the same instruction. The chosen launch
configuration, early thread completion, and divergent flow control can significantly lower the
number of active threads in a warp per cycle. This kernel achieves an average of 3.6 threads
being active per cycle. This is further reduced to 3.1 threads per warp due to predication. The
compiler may use predication to avoid an actual branch. Instead, all instructions are scheduled,
but a per-thread condition code or predicate controls which threads execute the instructions.
Try to avoid different execution paths within a warp when possible. In addition, ensure your
kernel makes use of Independent Thread Scheduling, which allows a warp to reconverge after a
data-dependent conditional block by explicitly calling __syncwarp().

Getting about 3 active threads per warp means Iâm only tapping into about 1/10th of the GPUâs
compute capacity, at best. Not exactly what you would assume from the 95% compute utilization figure
reported by nvidia-smi â that figure really just means Iâm doing something on all of the GPUâs
Streaming Multiprocessor or SM units about 95% of the time â even if that something is very
inefficient at using each SM, as in this case.

I highlighted the most relevant part of the remediation advice above, which is essentially to remove
the nested branching logic as much as I can. To do that in my program, where each thread is working
on a different game, I realised I would have to rewrite the core game play function. The original
version uses position within the codeâs nested branching structure to encode part of the game state
â I needed to replace that with an explicit data representation of all pieces of game state. Then
every step of the inner loop would be executed in unison across all threads playing their own games,
just with different data values.

Rewrite to use a lookup tableâ

To make the inner loop purely data driven, I chose to introduce a state transition table. Game state
is now tracked explicitly in variables which are treated as inputs to the transition lookup table,
followed by a set of actions that is always performed on every iteration using data values from the
lookup table.

A critical realization for implementing this cleanly in my case was to notice that the discard pile
can be treated as a third player in the game, with some extra states to track when the discard pile
is âplayingâ its cards to the player who won the last trick. With that mental twist in place, a
fairly straightforward lookup table became possible to write by hand. The code for this version is
here; it works on both CPU and GPU, but
it is slower on both â about half the speed of the baseline version on CPU, and two-thirds on
GPU. â¹

Being slower on CPU was expected, since it requires more instructions to manipulate more state
variables, and there is more memory access to lookup state transitions. We also have forced the
adding of the discard pile to the winning playerâs hand on every trick to be done slowly, as
individual turns with all the overhead needed for that. However, on GPU I had hoped to see gains
because now more threads in each warp should be able to execute in unison.

In fact Speed-of-Light compute and memory utilization figures did improve, to 17% and 38%
respectively, but Thread Divergence was still listed as a remediation warning. We are at least up
from an average of 3.6 active threads per warp to 5.3, but that is small comfort given the actual
speed is now about back to where I first started on GPU, which is well behind the CPU performance.

Ah, be careful about function exitsâ

What I had forgotten is that thread divergence will also occur when threads are exiting from a
nested function inside the kernel. My game play loop would always play one game (inside the function
play), then return to the search loop to swap two cards before calling play again. It basically
didnât matter that inside the play inner loop thread divergence has been largely solved, because
each thread in a warp will still finish at different times, depending on how many turns are needed
for their respective games. So they usually diverge at game end, and the whole warp of threads must
wait for the longest game amongst them to finish. Stats on the range of game lengths shows a minimum
of about 33 turns, an average of about 250 turns, and a max somewhere in the many thousands of turns
at least. Thatâs a lot of variation.

With that realization, my next program refactoring was to include the logic for game completion
book-keeping and switching to a new game as another (necessarily conditional) step in the inner game
playing loop. This slows down the inner loop even further, but allows the threads to stay mostly
converged across multiple games. To support this, I pre-create a backlog of games to play, ready for
the next game to be picked very quickly from inside the inner loop by whichever thread is available
first. (This introduces the only inter-thread synchronization mechanism needed so far, which is the
use of a CUDA atomic operation to read and increment the next_deal_to_play index, which barely
affects speed but solves the race condition.)

In order to allow the threads to run as long as possible in this synchronized inner loop, I decided
to use a big chunk of main GPU memory to hold a large backlog, which so far has barely been used
(just for the search loopâs best-game-so-far records).

We now get to a near-final version of the program, which can be recreated from the latest
version using some conditional
compilation definitions.
This version fills a large (eg. 1M entry) backlog of deals to play in GPU memory (using another
kernel of cooperating GPU threads working in parallel), which is then processed in parallel by 1024
threads (32 blocks of 32 threads). Performance did indeed improve over the initial version with the
lookup table, but only to about the same speed as the baseline version, which was achieved once
I had tweaked the number of blocks and threads. What gives?!

Ah, memory speed really mattersâ

Nsight Compute reveals Speed-of-Light is about the same as before (12% compute and 37% memory
utilization), and again lists a number of warnings. This time, some other warnings now seem more
relevant and actionable (as Iâve highlighted below):

[Warning] All pipelines are under-utilized. Either this kernel is very small or it doesnât issue
enough warps per scheduler. Check the Launch Statistics and Scheduler Statistics sections for
further details.

[Warning] Every scheduler is capable of issuing one instruction per cycle, but for this kernel
each scheduler only issues an instruction every 6.0 cycles. This might leave hardware resources
underutilized and may lead to less optimal performance. Out of the maximum of 8 warps per
scheduler, this kernel allocates an average of 1.00 active warps per scheduler, but only an
average of 0.17 warps were eligible per cycle. Eligible warps are the subset of active warps that
are ready to issue their next instruction. Every cycle with no eligible warp results in no
instruction being issued and the issue slot remains unused. To increase the number of eligible
warps either increase the number of active warps or reduce the time the active warps are
stalled.

[Warning] On average each warp of this kernel spends 2.4 cycles being stalled waiting for a
scoreboard dependency on a L1TEX (local, global, surface, texture) operation. This represents
about 39.7% of the total average of 6.0 cycles between issuing two instructions. To reduce the
number of cycles waiting on L1TEX data accesses verify the memory access patterns are optimal for
the target architecture, attempt to increase cache hit rates by increasing data locality or by
changing the cache configuration, and consider moving frequently used data to shared memory.

Thread Divergence is still listed as a warning, though the average active threads per warp is now at about 9. Looking at the âsourceâ view of the kernel profile, which shows per-instruction info such as the execution counts, average active threads, distribution of instruction stall reasons etc, it looks like there are lots of times when the inner loop instructions are stalled waiting for access to results from memory.

So it looks like there are two clear problems we can address:

Ensure there are more (warps of) threads available to schedule, so they can hide the
micro-latencies that are naturally experienced by any one warp.

Reduce the times when the inner loop is stalled accessing GPU memory.

The first issue can in theory be addressed by playing with the blocks and threads configuration.
However, changing that doesnât really make a difference: we are primarily bottlenecked on accessing
data from GPU memory.

Use Shared Memory as much as possibleâ

The best improvement we can make now is ironically to stop using main GPU memory as much as we can.
In my case, that means replacing the single large backlog of deals with a very small backlog per
block, held in Shared Memory.

As you may recall from CUDA Programming primers, Shared
Memory is the fastest memory area
(other than registers allocated by the compiler) available to programmers, but the most limited in
size. On my laptopâs GPU, it is limited to 48KB â the same amount of memory as my first home
computer (from the Apple II era). This memory is per-block and only available while each specific
block is executing â its contents will be lost when that block ends, so relevant info needs to be
copied to main memory as final output.

Thankfully for my program, using this only involves a modest code change, and basically means the
inner game loop runs for fewer iterations before the local backlog is exhausted and has to be
replenished. Also each small backlog is shared by fewer threads, so there is more opportunity for
some threads to finish early and increase thread divergence.

With this
change (which
included changing blocks and threads to 16 and 128, in order to try and provide more warps to the
scheduler as noted above), performance on GPU finally moved ahead of CPU for the first time, and by
a fairly impressive margin, reaching about 40M deals per second (vs. 3M on CPU)!

Nsight Compute is still saying we are memory bottlenecked though. Final chance to squeeze further!

Make Shared Memory stretch as far as possibleâ

My final change is to recognise that the core data structures are using more memory than they need
to, since Iâm using an enum to represent 5 possible card values, and enum by default is
represented as an int in C++, which is treated as a 32-bit word in CUDA. Similarly, the lookup
table data values are all defined as int, but far fewer bits would suffice. I had initially
wondered whether native word sized values were more efficient at the individual instruction level,
but actually GPUs (like most CPUs) efficiently support sub-word data sizes and even arbitrary
bit field operations at the instruction level, so that isnât an important concern.

With a change to specify uint8 as my enum base type, add appropriate bit field declarations in the
lookup table struct, and use a more compact representation of deals in the backlog (not the 64 entry
circular buffer representation used for playing fast), I am able to squeeze a longer backlog into
the 48KB of shared memory, and also reduce the memory bandwidth needed for lookup operations and
other game play steps.

The effect is rather gratifying: my final
version
now hits over 100M deals per second, at least until thermal throttling brings it back down to
95M or so. ð

Nsight Compute is still saying Iâm memory bound, but at this stage Iâm thinking that may be just
how it is. The algorithm is so light on computation that it will typically be waiting on memory
(even the super-fast Shared Memory) no matter what. The next step, if it were possible to see how to
restructure the game playing loop in a suitable way, would be to try and ensure we created coalesced
memory access patterns (where threads in a warp all reference directly adjacent memory locations
that allow the hardware to make them one read/write operation). But that seems very unlikely to be
possible with each thread still working on its own deal.

Maybe there is still some potential for improvement from tweaking block and thread counts, and
paying attention to memory layout and other micro-details. Iâm not holding my breath!

Progression Historyâ

From initial CPU version to final decent GPU version, here was the history of changes and results.
All performance numbers are in millions of deals per second.

Product

Use Cases

Pricing

Resources

About


Optimizing Sequential cuBLAS Calls for Matrix Operations—Alternatives to Kernel Fusion?

I am currently working on a CUDA project where my code involves a sequence of matrix multiplications followed by activation functions. Typically, such dependent, sequential operations can be optimized using kernel fusion to minimize shared memory access, enhancing overall performance.

To streamline my implementation, I opted to use cuBLAS for handling the matrix multiplications. However, I’ve found that cuBLAS doesn’t support kernel fusion, which seems like a missed opportunity for optimization in terms of reducing memory overhead and improving execution speed.

Given this context, I am seeking advice on alternative methods to optimize these sequential cuBLAS calls. Are there techniques within CUDA or associated libraries that can mimic the effects of kernel fusion, or perhaps a way to efficiently manage these operations to achieve similar performance gains? Any suggestions on optimizing memory usage or overlapping computations would also be greatly appreciated.

Share this problem. Way I understand it cuBLASDx aims to facilitate kernel fusion for BLAS operations. But currently it looks to only support matrix multiplication which is not enough in my case.

You can use cublasLT for some fusion cases, or directly using cutlass.

Please take a look at cuDNN’s Graph API (Graph API — NVIDIA cuDNN v9.1.0 documentation). It supports fusion prologue and epilogue fusions with convolutions and matmuls. It offers an abstraction layer on top of cublas and cutlass.

Generic Runtime Fusion Engines (Graph API — NVIDIA cuDNN v9.1.0 documentation)

Related topics

Powered by Discourse, best viewed with JavaScript enabled


Democratizing AI Accelerators and GPU Kernel Programming using Triton

by Sanjeev Rampal | Nov 7, 2024 | AI, Hybrid Cloud

Red Hat’s Emerging Technologies blog includes posts that discuss technologies that are under active development in upstream open source communities and at Red Hat. We believe in sharing early and often the things we’re working on, but we want to note that unless otherwise stated the technologies and how-tos shared here aren’t part of supported products, nor promised to be in the future.

Triton1 is a language and compiler for parallel programming. Specifically it is currently a Python-based DSL (Domain Specific Language) along with associated tooling, that enables the writing of efficient custom compute kernels used for implementing DNNs (Deep Neural Networks) and LLMs (Large Language Models), especially when executed on AI accelerators such as GPUs.

The key goals for Triton are:

Consequently Triton aims to democratize AI infrastructure, accelerate data science developer productivity (i.e., “developer inner loop”), enabling an open architecture for GPU and AI accelerator programming.

The Triton project is currently released as open source by Philippe Tillet and OpenAI under an MIT license (with growing contributions from Meta and others). Red Hat is a strong proponent of Open Source AI technologies and innovations that facilitate a healthy and diverse hardware ecosystem that lowers costs and expands adoption of AI infrastructure solutions. In this blog post, we describe some of the foundational architecture topics in the Triton space and its connection with frameworks such as PyTorch. In subsequent blog posts, we will go into further details including the use of Triton on Red Hat platforms.

Kernels, GPU programming models

Figure 1 below illustrates a simplified view of a basic AI server model with a single multi-core CPU and a single GPU. A GPU kernel is simply the program/ function that runs on the GPU, loaded and invoked on-demand from the program running on the host CPU. A common GPU architecture is that of a SIMD/ SPMD machine that itself contains a number of processors (often referred to as Streaming Multiprocessors or SMs), which then themselves contain multiple smaller compute cores as well as specialized arithmetic units such as Tensor cores.

When running a DNN application, the conventional design pattern is that the host CPU launches new GPU kernels onto the GPU, loads a (often very large) set of data into GPU memory, lets the GPU processors execute multiple parallel compute threads on the loaded kernel to perform a set of computations, (usually vector or matrix operations such as matrix multiplications) and then harvests the results back into CPU memory before potentially launching a follow up set of kernels and associated data. Later in this article we see how newer design patterns optimize this. A precise comparison of the vector processing differences between CPUs and GPUs is beyond the scope of this article, save to mention here that general purpose CPUs typically contains 10s or maybe 100s of general purpose compute cores with some amount of vector processing support, whereas GPUs contain 1 or 2 orders of magnitude more special purpose compute threads (e.g., “CUDA threads”) enabling massively parallel processing on large vectors in an SIMD manner and also have special purpose tensor arithmetic units e.g., “tensor cores”.

Figure 2 (Ref. article) shows a common abstracted model of a single GPU that can serve for the purpose of this article. This shows that there is a multi-level memory hierarchy even within a single GPU, with an L1 SRAM based cache available to compute threads within a single SM, an L2 SRAM based cache shared by multiple SMs and a global GPU memory (often implemented using a form of DRAM called HBM or High Bandwidth Memory). The SRAM memories support higher throughput and significantly lower access latencies than the global HBM which in turn is faster than CPU DRAM. The exact numbers vary with GPU type but as an example on the H100, HBM memory bandwidth is 3 TB/s, L2 SRAM bandwidth is 12 TB/s and L1 SRAM bandwidth is 33 TB/s.

Newer GPUs continue to add special capabilities that are not shown in the above simplified model, an example being the Tensor Memory Accelerator or TMA. A detailed analysis of the implications of these different memory hierarchies as well as the memory and compute bandwidths of these different components is beyond the scope of this article. The key point to note is that in order to achieve high efficiency and performance for DNNs, it is vital that all the compute cores, SMs and tensor cores of the GPU are kept busy, particularly with large transformer style models. See for example GPUs go brr. Due to advances in GPU hardware design technology, the raw compute capacity of GPU SMs have vastly outpaced the improvements in memory bandwidth implying that the arithmetic cores are often idling if data is not already in the GPU’s SRAM or if there isn’t enough computation designed in per HBM memory fetch.

Due to the mismatch between throughputs of the various compute and memory components, a naively written AI application might well utilize the GPU’s SMs and tensor cores at less than 10% utilization (or sometimes even less than 1% utilization) resulting in both poor overall performance due to increased latency and overall execution time as well as high cost of providing the service given that these expensive GPUs are being utilized at a tiny fraction of their potential compute capacity. See for example “AI and memory wall”. Hence the performance of transformer model based applications is often said to be “memory-bound”, although there can be some scenarios where the performance is “compute-bound” as well.

This is where a well designed GPU kernel can make a huge difference and improve overall performance and efficiency. Industry and academic research in this area has emphasized the need to design kernels that are better tuned to address this compute vs memory performance imbalance. See for example “Data movement is all you need” and “GPUs go brr”.

Having learnt a bit about GPU architecture and the value of well written and tuned GPU kernels, we now come to what types of kernels one could have.  CUDA kernels are popular examples of such kernels that are specific to GPUs and AI accelerator hardware from one vendor. Similarly vendors of other AI chips have their own kernel stacks. For instance AMD has open sourced its software platform called ROCm for building compute kernels for its AI accelerators and GPUs.

Triton

With this background perspective, we now come to the Triton kernels.  Triton is a DSL for writing GPU kernels that aren’t tied to any one vendor. Additionally, it is architected to have multiple layers of compilers that automate the optimization and tuning needed to address the memory vs compute throughput tradeoff noted above without requiring the kernel developer to do so. This is the primary value of Triton. Using this combination of device-independent front-end compilation and device-dependent backend compilation, it is able to generate near-optimal code for multiple hardware targets ranging from existing accelerators to upcoming accelerator families from Intel, Qualcomm, Meta, Microsoft and more without requiring the kernel developer to know details of or design optimization strategies for each hardware architecture separately.

Effectively, this moves the detailed and often vendor-specific complexity out of the user’s concern and into the vendor’s.  This is a more logical design split because vendors are incentivized to highlight their hardware’s differentiating features, and user’s immediately benefit without change to their applications and do not have to build a depth of engineering proficiency in GPU programming for each model GPU they own. However we are still in an early phase of this technology and should monitor how back-end compiler tuning performance evolves.

Figure 3 below illustrates some of the different ways that Triton could fit into a DNN software stack in combination with a PyTorch framework. Triton can also be used in combination with other frameworks such as TensorFlow.

As shown, a DNN application may choose to make direct calls to launch custom Triton kernels (either developer authored or leveraged from open source Triton kernel library repos) to perform certain GPU intensive functions. Alternatively, it may choose to leverage a framework such as PyTorch. Within the PyTorch community, additional frameworks have been recently introduced as part of PyTorch 2.x which can further automate the options for using Triton kernels. For instance by using the Torch Dynamo and Torch Inductor compiler frameworks, Triton kernels can be automatically generated and optimized (including optimizations such as kernel fusions) without the app developer having to manually write, fuse or invoke an existing library of Triton kernels directly. However not all kinds of kernel functions can be automatically generated in this manner with optimal performance, so a mix of generated and manually developed Triton kernels may be used in practice. Several industry and open source projects already exist to develop well designed Triton kernels for various DNN functions. Examples include. LinkedIn’s Liger, Unsloth and sample kernels from the Triton team. We haven’t listed example Triton kernels in this article in order to focus on architecture and analysis. The official Triton repo has some tutorials with kernel examples and explanations that may be referred to.

In any case, once we have the set of well designed Triton kernels, the Triton language compiler performs multiple passes and its own set of optimizations over the kernel code to generate a Triton IR/ Intermediate Representation version of the code. Beyond this point, we enter the realm of device and GPU specific compilations where backend compilers, typically provided by GPU vendors, translate the lower versions of the kernels into GPU specific machine code binaries for eventual execution on the hardware runtimes. Although we have skipped over many of the details, it should be clear to the reader that the overall process involves multiple layers and passes of compilations, code generation and optimizations when going from a high level DNN application all the way to binary code executable on specific hardware devices and GPUs.

Definitions of some key concepts

The prior sections provided some intuition around design issues relevant to Triton, GPU kernels and frameworks such as Pytorch. For completeness it is useful to have a short reference list below of common terminology which the reader may find handy when further researching the literature in this area as well as in advance of our future blog posts on these topics.

Kernel Fusion – Optimization technique of combining multiple compute operations/ kernels into a single GPU kernel, optimizing memory transfers, minimizing latency and improving overall performance by executing related operations together in a single pass on the GPU.

Auto-tuning – A process of automatically optimizing parameters (such as block size, thread count) to find the best-performing configuration for a specific hardware setup and workload, improving execution efficiency.

Arithmetic Intensity – The ratio of computational operations (e.g., floating-point operations) to memory operations (data transfers), indicating how much work is done per memory access.

Conclusion

Triton is an important initiative in the move towards democratizing the use and programming of AI accelerators such as GPUs for Deep Neural Networks. In this article we shared some foundational concepts around this project. In future articles, we will dive into additional Triton details and illustrate its use in enterprise AI platforms from Red Hat.

Acknowledgements: Thanks to Steve Royer (Red Hat), Jeremy Eder (Red Hat), Raffaele Spazzoli (Red Hat), Adnan Aziz (Meta), Adam Goucher (OpenAI), Madhukar Srivatsa (IBM) and Raghu Ganti (IBM) for their valued input and review.

Explore

Privacy statement

Terms of use

All policies and guidelines

About

Copyright © 2021-2023 Red Hat, Inc.