Reasoning about performance from first principles

When trying to optimize performance on a computer, we can simplify the hardware into two things [1]

For the RTX 3090 this is around 35.5 TFLOPs with a max global memory bandwidth of 936 GB/s. We can idealize the program running on the computer as loading K bytes and performing N operations with each byte, giving us the arithmetic intensity. When trying to understand what the upper bound of tasks that can be performed per second, we can use the equation below.

$$P = \min\left(\frac{\text{flop}}{s}, \frac{N}{K} \times \frac{\text{bytes}}{s}\right)$$

By multiplying the memory bandwidth by the arithmetic intensity we adjust for the fact that each loaded byte results in K operations. The resulting number P is the minimum of the memory bandwidth (adjusted for arithmetic intensity) and the theoeretical FLOPS/s. This tells us the upper bound for performance as well as whether the program will be compute-bound or memory-bound. This beautiful abstraction is true for any machine based on the Von Neumann architecture, not just GPUs.

â

â

The plot above shows a blue line, which is AI * Mem-Bandwidth, as well as a FLOPS/s line which is horizontal. The point at which the blue line crosses the red-line shows what the arithmetic intensity of a program would need to be to fully saturate the 3090s compute units. We need to perform ~38 operations per loaded byte to get to this point! Lets look at arithmetic intensity for vector addition/matrix multiplication, and quantify their AI.

â

FP32 Vector Addition (4 bytes per element)

1) Load \(4 N\) bytes for Vector A.

2) Load \(4 N\) bytes for Vector B.

2) Perform ~\(N\) Â FP32 add operations to add Vector AÂ &Â B.

3) Store \(4 N\) bytes.

Arithmetic Intensity (AI) is \(\frac{N}{12N}\) operations/bytes or 0.0833.

Vector addition is heavily memory bound, since the arithmetic intensity is so low.

â

FP32 Matrix Multiplication (all matrix dimensions `N` for simplicity)

1) Load \(4 N^2\) bytes for Matrix A.

2) Load \(4 N^2\) bytes for Matrix B.

3) Perform \(2 N^3\) operations (there are \(N^2\) outputs, each output requires a dot-product of vectors with size \(N\), each dot-product requires ~\(2 N\) additions & multiplies).

3) Store \(4 N^2\) bytes.

Arithmetic Intensity (AI) is `\(\frac{2 N^3}{12 N^2} = 0.167 N\) ops/bytes.

Matrix multiplication is interesting because its AI is a linear function of the sizes of the input matrices. As a result, the program is memory bound at smaller sizes but becomes compute bound at larger ones.

â

This analysis gives us the âlight-speedâ for a given program. In reality, we won't load at theoretical bandwidth, we won't compute at theroetical max FLOPs, we may perform more than the ideal number of operations specified during program execution, there will be wind up/wind down effects, etc. By comparing an actual kernelâs effective ops/s to the idealized we can invest our efforts in tackling the right bottle-neck, and better understand how far we are from theoretical limits.

Memory Optimizations

The optimizations below are some of the most important for getting good performance. If the compute units arenât getting a high throughput stream of bytes to crunch on, the fact that the GPU has an absurd number of compute units wonât matter. [2]

Coalesced and Aligned Global Memory Access [3]

When accessing global memory always do your best to have each thread access sequential addresses where the array is aligned, ideally to 128B (using cuda malloc will default align to 256B). When an RTX 3090 performs a cached GMEM load, it will pull in cache-lines of size 128byte. If a warp (32 threads) accesses 32-bit addresses the entire load can be processed in a single transaction. Deviating from the sequential nature of the access, or misaligning memory in GMEM during kernel launch will reduce effective GMEM utilization.

As we learned in the intro post, unaligned access can either cause

a) excessive pre-charging of row-buffers due to having to pull in multiple DRAM rows

b) excessive memory controller overhead from having to pull multiple cache-lines within the same DRAM row

Both lead to unnecessary overhead. If strided access is necessary, use shared memory as an intermediary to allowing for coalescing.

Use Vectorized Memory Access Instructions [4]

Lines issuing memory loads in CUDA typically compile to a 32-bit (one word) load using the LD.E/ST.E instructions in SASS. If you know the thread will require multiple sequential words of memory, and have a piece of data that is aligned in memory to that multiple, you can issue vectorized instructions to load multiple words in a single transaction. Loading in this way reduces instruction overhead and can be combined with coalescing to improve memory throughput/latency. Vectorized loads are accomplished by using vector data types (float4, float2, int4, int2, etc) and typecasting. An example of a kernel that uses vectorizes loading of two consecutive 32-bit integers is shown below.

â

__global__ void device_copy_vector2_kernel(int* d_in, int* d_out, int N) {
  int idx = blockIdx.x * blockDim.x + threadIdx.x;
  if (idx < N / 2) {
    reinterpret_cast<int2*>(d_out)[idx] = reinterpret_cast<int2*>(d_in)[idx];
  }

  // Only one thread processes the final element, if N is odd
  if (idx == 0 && N % 2 == 1) {
    d_out[N - 1] = d_in[N - 1];
  }
}

â

Avoid Shared Memory Bank Conflicts

Shared memory is divided into 32 banks of SRAM cells, with a controller for each bank that can serve 1byte/clock cycle. When multiple threads access a bank simultaneously, these accesses will serialize. The easiest way to avoid shared memory bank conflicts is to access sequential shared memory addresses with each thread within a warp (similar to coalesced GMEM access).

If assigning each thread to a separate bank isnât feasible, memory padding can be used to introduce an offset that eliminates the conflict. The padding is at the cost of higher shared memory utilization per block which may impact occupancy.

Keep Re-Used Data in the fastest Memory Stores

Many operations involve data re-use. When performing a matrix multiplication for example, we load 2N^2 elements, but perform 2N^3 operations, meaning each element has N operations associated with it. It would be extremely wasteful to go back to global memory for each of the N operations. When performing operations with the same data multiple times, consider keeping it in shared memory or thread registers. Simon Boehm's post on optimizing GEMM makes heavy use of this method and I would highly reccomend giving it a thorough read. NVIDIA GPUs also have constant and texture memory, which to be totally honest I have not used. But from what I understand, these stores are read-only and can provide efficient access if many threads will need to access the same memory address many times [7]. When initially thinking about kernel architecture, spend a fair amount of time understanding the graph of memory dependencies for each output element that your kernel produces. Visualizing this graph can provide a clearer picture of what bytes should be put where.

Avoid Register Spilling [8]

GPUs have a maximum number of registers that can be allocated to each thread. When a thread violates this limit (255 for RTX 3090), the data will spill to âlocal memoryâ which is actually just a section global memory set aside for that thread! If we are lucky, L2 cache will intercept the read/writes and we wonât have to pay full latency associated with GMEM. But if we are not careful, we can inadvertently perform tons of slow GMEM accesses and delude ourselves into thinking we are using fast thread registers. When compiling a CUDA kernel, you can add â-Xptxas -vâ to your nvcc command to see a print out of register use per thread and make sure you arenât close to any limits.

â

Compute Optimizations

Getting bytes to compute units efficiently is important, and so is make sure the compute units themself are adequately saturated with arithmetic instructions that operate on the incoming bytes. [2]

Maximize Number of Active Warps (Occupancy) [9]

I mentioned in the previous post that its up to us to make sure all of compute units are executing useful operations during the course of the kernel execution. This can be done by thinking carefully about memory resources and thread count in each block, since this is the fundamental limiter to how much warps can be active at the same time. On an RTX 3090, each SM can support:

â

â

Notice that while the number of max active threads is 1536, there are only 128 CUDA cores on an SM. Whats happening here is the warp scheduler tries to make as many warps âactiveâ as possible by assigning them the registers and shared memory they request. When warps are stalled due to memory latency, the scheduler can swap active threads in and out of CUDA cores to make sure the hardware is fully utilized. This is what occupancy measures: how many of the maximum possible active warps are able to be added to the pool of warps ready to run on a core? In this way the GPU hides memory latency by oversubscribing the hardware. The GPU can only oversubscribe if its fed kernels that make availiable blocks with the appropriate number of threads/warps, and donât hog all the registers/shared memory. While low occupancy will certainly hurt performance, high occupancy doesnât guarante compute unit saturation. According to this post on NVIDIA forums, 50% - 75% occupancy is usually acceptable. Also note that while per SM occupancy is important, we also want to saturate all SMs with work. There are 84 SMs on an RTX 3090 so we need atleast 84 blocks to make sure each has something to do.

â

Tangent

I havenât tried this out myself but I do think going forward it would be best to write kernels that are as agnostic to register/shared-mem use/block dimension as possible in order to let an auto-tuner figure out what allocation of resources is optimal for performance. Sometimes it can be better to have fewer threads per block and more work per thread. This opens up more thread-level ILP (instruction level parallelism) and can enable more thread registers per thread. This is particularly true when performing block-level reductions as fewer threads means less thread â thread comms. This could lead to lower occupancy but better over all performance.

Use Tensor Cores & FMA Units [14] [15]

Tensor cores are designed specficially to accelerate matrix-multipy-accumulate operations on GPUs. Use them whenever an operation can be represented as an MMA. On a similar grain of thought, scalar-multiply-accumulates are also hardware accelerated and can be called using the fmaf() function in CUDA. The NVCC compiler typically optimizes operations of the format âa = a + (b*c)â into FMA instructions anyway but using the function call can make this explicit. One thing to keep in mind though, is they donât benefit memory-bound workloads. For example, a convolution can be performed as an implicit GEMM in order to utilize tensor cores, but the memory overhead of the transforms needed to achieve this may far outweigh efficiency gains from tensor core utilization for low arithmetic intensity workloads. Donât worry if this statement is confusing for now, future posts go into roof-line models and their implications.

Minimize Warp Divergence [10]

When each thread in a warp of 32 threads executes, control flow overhead is minimized when each thread is performing the exact same operation as the same time. Certain types of data-dependent control flow can cause this to no longer be the case. In these situations, the threads will effectively split into multiple diverged execution paths, with each chunk executing independent of each other. Obviously this hurts hardware utilization as we are running fewer than 32 threads per warp. Try to minimize warp divergence to the extent possible by making sure all threads in a warp will follow the same execution path.

Unroll Loops [11]

When CUDA code is compiling to PTX, loops with loop counts that are defined at compiled time will get unrolled. Unrolling eliminates overhead associated with checking loop conditionals/incrementing loop count and enables instruction level parallelism in the case of loops that unroll to multiple indepedent instructions. Lets take a look an example with a simple for-loop. We can hint to the compiler to unroll a loop by using â#pragma unrollâ. Appending an integer after pragra unroll tells the compiler how many to iterations to unroll. By putting a 1 after pragma unroll we can effectively prevent the compiler from unrolling the loop.

Standard Loop CUDAâ

float temp = data[idx];
#pragma unroll 1
for (int i = 0; i < 10; ++i) {
    temp += i;
}
data[idx] = temp;

â

Standard Loop PTX

$L__BB0_1: Â  Â  Â  Â cvt.rn.f32.s32 Â %f4, %r5; //convert value in r5 to float and move to f4 Â  Â  Â  Â add.f32 Â  Â  Â  Â  %f5, %f5, %f4; //add value from f4 to f5 and store in f5 Â  Â  Â  Â add.s32 Â  Â  Â  Â  %r5, %r5, 1; //increment loop counter in r5 by 1 Â  Â  Â  Â setp.ne.s32 Â  Â  %p1, %r5, 10; //compare loop counter (r5) to 10 and set predicate register p1 to True if it is  Â  Â  Â  Â @%p1 bra Â  Â  Â  Â $L__BB0_1; //branch conditionally based on the value in p1 Â  Â  Â  Â st.global.f32 Â  [%rd1], %f5; Â  Â  Â  Â ret;

Unrolled Loop PTX (CUDA code uses #pragma unroll instead of #pragma unroll 1)

add.f32 Â  Â  Â  Â  %f2, %f1, 0f00000000;add.f32 Â  Â  Â  Â  %f3, %f2, 0f3F800000;add.f32 Â  Â  Â  Â  %f4, %f3, 0f40000000;add.f32 Â  Â  Â  Â  %f5, %f4, 0f40400000;add.f32 Â  Â  Â  Â  %f6, %f5, 0f40800000;add.f32 Â  Â  Â  Â  %f7, %f6, 0f40A00000;add.f32 Â  Â  Â  Â  %f8, %f7, 0f40C00000;add.f32 Â  Â  Â  Â  %f9, %f8, 0f40E00000;add.f32 Â  Â  Â  Â  %f10, %f9, 0f41000000;add.f32 Â  Â  Â  Â  %f11, %f10, 0f41100000;st.global.f32 Â  [%rd4], %f11;ret;

Note that in the unrolled case the compiler turned the loop into 10 distinct add.f32 instructions, and derived the constant values based on the loop count at compile time. This change eliminates loop related overhead. In this case there is a dependency between each instruction but in some cases independent instructions resulting from unrolling can also allow the thread to utilize greater instruction level parallelism. The CUDA compiler is quite good at spotting loops that can be unrolled but throw in a â#pragma unrollâ for loops you think could benefit from unrolling.

Use Signed-Ints for Loop Counters [12]

Unsigned integers have defined overflow behavior as they are expected to loop back around, whereas signed ints result in undefined behavior at runtime. Ensuring the former holds true reduces the compilerâs ability to optimize loop execution. As a result you may see a small pef improvement in hot-loops by using signed-ints for loop counters.

Use Fast Math Library (when precision isnât critical) [13]

CUDA provides a pretty extensive math library for operations that execute on the special functions unit. Some examples of functions include - sin(x), cos(x), log(x), exp(x), etc. If you donât care as much about precision and can accept some rounding errors (SIDE BAR ABOUT HOW INT8 works so rounding errs prob fine), using the fast version of these calls can improve performance. Examples - __sinf(x), __cosf(x) vs. sin(x), cos(x)

Maximize Instruction Level Parallelism via Dual-Issue Instruction Dispatch

According to discussion in this thread, dual-instruction dispatch on NVIDIA GPUs isnât a huge driver of improved performance and not worth too much thought. But it is worth noting that the warp scheduler can issue up to two instructions per cycle IF the are multiple instructions with no data or control flow dependencies. Writing a kernel such that there are fewer dependencies and diversity in the types of execution units being used (FP32/tensor core/load-store units/etc) may enable higher instruction dispatch per cycle.

â

References

[1] https://people.eecs.berkeley.edu/~kubitron/cs252/handouts/papers/RooflineVyNoYellow.pdf

[2] https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/contents.html

[3] https://developer.nvidia.com/blog/how-access-global-memory-efficiently-cuda-c-kernels/

[4] https://developer.nvidia.com/blog/cuda-pro-tip-increase-performance-with-vectorized-memory-access/

[5] http://homepages.math.uic.edu/~jan/mcs572f16/mcs572notes/lec35.html

[6] https://slideplayer.com/slide/12553635/

[7] https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#constant-memory

[8] https://developer.download.nvidia.com/CUDA/training/register_spilling.pdf

[9] https://on-demand.gputechconf.com/gtc-express/2011/presentations/cuda_webinars_WarpsAndOccupancy.pdf

[10] https://people.maths.ox.ac.uk/gilesm/cuda/lecs/lec3-2x2.pdf

[11] https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#branch-predication

[12] https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#loop-counters-signed-vs-unsigned

[13] https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#math-libraries

[14] https://developer.nvidia.com/blog/programming-tensor-cores-cuda-9/

[15] https://forums.developer.nvidia.com/t/fma/32965


OpenCL Kernel Memory Optimization - Local vs. Global Memory

Hi,

I’m new to OpenCL and I consider using it for some graphics computation where using an OpenGL shader seems not to be natural. Before I actually do so I thought I’d try how much of a performance improvement I could get using OpenCL on my Nvidia GTX 460 over my CPU. For this reason, I implemented a simple skeleton skinning algorithm, once on the CPU, without multithreading but using the Eigen library, which provides SSE-optimized vector and matrix libraries, and once in an OpenCL kernel executing on the GPU. The vertices, bone matrices etc. are generated randomly on application start. I repeat the whole skinning several times so that it executes long enough to get meaningful timing results.

First I simply tried a kernel where I have as much work-items as I have vertices, each one generating one output vertex. I quickly saw that this is not a good idea because performance was even worse than on the CPU. I figured this was in essence a problem of too many memory accesses, mainly to the bone matrices, which are an array of float16-vectors that is addressed four times in each work-item. Then I changed the algorithm so that each work-item handles multiple output vertices, one after the other, so that I have less work-items. In each work-group I create a copy of the bone matrices in local space, and further accesses to these matrices come from local space. The interesting part of my C++ code looks like this:

#define NUM_BONES 30
#define NUM_VERTICES 30000
#define NUM_VERTICES_PER_WORK_ITEM 100
#define NUM_ANIM_REPEAT 1000

uint64_t PerformOpenCLSkeletalAnimation(Matrix4* boneMats, Vector4* vertices, float* weights, uint32_t* indices, Vector4* resVertices)
{
    File kernelFile("/home/alemariusnexus/test/skelanim.cl");

    char opts[256];
    sprintf(opts, "-D NUM_VERTICES=%u -D NUM_REPEAT=%u -D NUM_BONES=%u -D NUM_VERTICES_PER_WORK_ITEM=%u", NUM_VERTICES, NUM_ANIM_REPEAT, NUM_BONES, NUM_VERTICES_PER_WORK_ITEM);

    cl_program prog = BuildOpenCLProgram(kernelFile, opts);

    cl_kernel kernel = clCreateKernel(prog, "skelanim", NULL);

    cl_mem boneMatBuf = clCreateBuffer(ctx, CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR, NUM_BONES*sizeof(Matrix4), boneMats, NULL);
    cl_mem vertexBuf = clCreateBuffer(ctx, CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR, NUM_VERTICES*sizeof(Vector4), vertices, NULL);
    cl_mem weightBuf = clCreateBuffer(ctx, CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR, NUM_VERTICES*4*sizeof(float), weights, NULL);
    cl_mem indexBuf = clCreateBuffer(ctx, CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR, NUM_VERTICES*4*sizeof(uint32_t), indices, NULL);
    cl_mem resVertexBuf = clCreateBuffer(ctx, CL_MEM_WRITE_ONLY, NUM_VERTICES*sizeof(Vector4), NULL, NULL);

    uint64_t s, e;
    s = GetTickcount();

    clSetKernelArg(kernel, 0, sizeof(cl_mem), &boneMatBuf);
    clSetKernelArg(kernel, 1, sizeof(cl_mem), &vertexBuf);
    clSetKernelArg(kernel, 2, sizeof(cl_mem), &weightBuf);
    clSetKernelArg(kernel, 3, sizeof(cl_mem), &indexBuf);
    clSetKernelArg(kernel, 4, sizeof(cl_mem), &resVertexBuf);

    size_t globalWorkSize[] = { NUM_VERTICES / NUM_VERTICES_PER_WORK_ITEM };
    size_t localWorkSize[] = { NUM_BONES };

    for (size_t i = 0 ; i < NUM_ANIM_REPEAT ; i++) {
        clEnqueueNDRangeKernel(cq, kernel, 1, NULL, globalWorkSize, localWorkSize, 0, NULL, NULL);
    }

    clEnqueueReadBuffer(cq, resVertexBuf, CL_TRUE, 0, NUM_VERTICES*sizeof(Vector4), resVertices, 0, NULL, NULL);

    e = GetTickcount();

    return e-s;
}

The associated program/kernel looks like this:

inline float4 MultiplyMatrixVector(float16 m, float4 v)
{
    return (float4) (
        dot(m.s048C, v),
        dot(m.s159D, v),
        dot(m.s26AE, v),
        dot(m.s37BF, v)
    );
}

kernel void skelanim(global const float16* boneMats, global const float4* vertices, global const float4* weights, global const uint4* indices, global float4* resVertices)
{
    int gid = get_global_id(0);
    int lid = get_local_id(0);

    local float16 lBoneMats[NUM_BONES];
    lBoneMats[lid] = boneMats[lid];

    barrier(CLK_LOCAL_MEM_FENCE);

    for (int i = 0 ; i < NUM_VERTICES_PER_WORK_ITEM ; i++) {
        int vidx = gid*NUM_VERTICES_PER_WORK_ITEM + i;

        float4 vertex = vertices[vidx];
        float4 w = weights[vidx];
        uint4 idx = indices[vidx];

        resVertices[vidx] = (MultiplyMatrixVector(lBoneMats[idx.x], vertex * w.x)
                + MultiplyMatrixVector(lBoneMats[idx.y], vertex * w.y)
                + MultiplyMatrixVector(lBoneMats[idx.z], vertex * w.z)
                + MultiplyMatrixVector(lBoneMats[idx.w], vertex * w.w));
    }
}

Now, per work-item I have only one access to the global boneMats, when I create the local copy, and it’s even a lot less work-items executing altogether. Then I have NUM_VERTICES_PER_WORK_ITEM*4 accesses to the local array afterwards. As I understand, local memory should be way faster than global memory, so I thought this would greatly improve performance. Well, the opposite is the cause: When I let lBoneMats alias to the global boneMats instead, I get actually better performance than with the kernel listed above.

What did I get wrong here?

Thanks in advance!

I have still not found a solution for this problem. Does nobody have any idea, or anything I could try?

There is the Nvidia Visual Compute Profiler which can tell you some performance information.

Looking at your launch parameters, you are launching 300 work items arranged into 10 groups of 30 work items each.  On Nvidia GPUs, threads are grouped into warps - a group of 32 threads.  Each multi processor executes hundreds of threads.  Such a large number of threads are needed to hide the latency involved in accessing either global or local memory (although local memory accesses are not as costly).

There are several multi-processors, hence you need thousands of threads to adequately use a GPU.  That is why your global memory version is faster I think.  I would suggest changing your launch settings so that there are more threads (keep the shared memory though, just do less work per work item).  Also, use multiples of 32 threads.

a couple of things:

a) even if the gpu is slower than a cpu, not having to copy the data to/from the device could help. (i.e. go straight from opencl to opengl buffer or whatever)
b) looks like your loop is accessing memory (vidx) in pretty much worst-case access pattern.  each work-item should access adjacent values where possible.

To me it looks like it would be best implemented as a one-work-item-per-output algorithm as you said you first tried.  Or even use 4 work items per result (one for each of idx.xyzw).  Assuming the indexing is correct (i.e. set vidx == get_global_id(0)), i would have expected that to be faster.

I think you’re also confusing what local work size is - it is just the modulo of the total work size which is allocated to a given work-unit (i.e. shares LDS and some other resources).  It isn’t a separate dimension from global work size.  It is only a ‘coincidence’ that your code is working and num_vertices/num_vertices_per_work_item is a multiple of num_bones.

LDS is way faster than uncached global memory, but if you’ve only accessing 30 ‘bones’, then it should fit into L1 cache, in which case LDS isn’t that much of a boost (it depends on the hardware, not sure what it is on nvidia).

There is the Nvidia Visual Compute Profiler which can tell you some performance information.

I have tried it with CUDA 4.2, but couldn’t really see what it was trying to tell me. With CUDA 5.0 I can’t get OpenCL profiling to work at all. As I’ve read, OpenCL profiling seems to be broken in 5.0, and the driver of 4.2 does not compile on my 3.5 kernel, so I guess I have to wait until Nvidia fixes that (I’m sceptic as to whether they will at all).

I would suggest changing your launch settings so that there are more threads

Seems like this was the main problem. I have no idea what I do now that I haven’t done with my first implementation, but now it’s about four times faster than on my CPU+SSE.

Also, use multiples of 32 threads.

I don’t really know how to do that with my algorithm, as I have no way of controlling the number of vertices, but when I launch one thread for every vertex as I do now, I guess the bit of processing time wasted is not really significant anymore.

even if the gpu is slower than a cpu, not having to copy the data to/from the device could help. (i.e. go straight from opencl to opengl buffer or whatever)

Maybe I’ll do this in my final implementation.

looks like your loop is accessing memory (vidx) in pretty much worst-case access pattern. each work-item should access adjacent values where possible.

I can’t see any nice way to do this. Maybe presorting the vertices by their bone matrix indices, but that would be quite costly (although it would have to be done only once) and I don’t like the idea of changing the vertex order.

To me it looks like it would be best implemented as a one-work-item-per-output algorithm as you said you first tried. Or even use 4 work items per result (one for each of idx.xyzw). Assuming the indexing is correct (i.e. set vidx == get_global_id(0)), i would have expected that to be faster.

As I mentioned before, I do this now, and for whatever reason it’s faster than what I first tried.

I think you’re also confusing what local work size is - it is just the modulo of the total work size which is allocated to a given work-unit (i.e. shares LDS and some other resources). It isn’t a separate dimension from global work size. It is only a ‘coincidence’ that your code is working and num_vertices/num_vertices_per_work_item is a multiple of num_bones.

That’s what I thought it is: The number of work-items (threads) per work-group (thread block). I know I have to choose execution parameters so that the total number of work-items is evenly dividable by it.
In my understanding, changing local work size should not affect performance, assuming shared memory is not used (otherwise the more work groups you have, the more global-to-shared memory copies have to be done, assuming every work group always copies the same amount of data) and it is still a multiple of the warp size (because otherwise the warps aren’t fully utilized).

One question I still have which I couldn’t guess from Nvidias docs is: Can a single warp be made up of threads from different work groups (thread blocks)?

Although there might be room for further improvement, at least I can see that the GPU is actually faster than the CPU, so I’m satisfied for now. The only thing I can’t quite guess is why the same program runs about 8 times slower than the CPU on my old GeForce 8200, even when I optimize the execution parameters. I guess that is because it’s an onboard GPU and global memory accesses are even slower than on a GPU with dedicated memory. The same is true when I execute the CL program on my CPU device, but it might just be too massively multithreaded for a CPU, I haven’t tested this enough yet.

Anyway, thanks for your help!

That’s what I thought it is: The number of work-items (threads) per work-group (thread block). I know I have to choose execution parameters so that the total number of work-items is evenly dividable by it.
In my understanding, changing local work size should not affect performance, assuming shared memory is not used (otherwise the more work groups you have, the more global-to-shared memory copies have to be done, assuming every work group always copies the same amount of data) and it is still a multiple of the warp size (because otherwise the warps aren’t fully utilized).

Changing local work size will affect performance outside of just using LDS for a bunch of reasons: everything in the workgroup executes in lock-step, which affects cache and branching stuff, it affects how many registers are required which affects how many workgroups can be executed concurrently, etc.

BTW use a worksize multiple of 64 if you also want it to work well on AMD hardware, as that is the minimum it requires.

One question I still have which I couldn’t guess from Nvidias docs is: Can a single warp be made up of threads from different work groups (thread blocks)?

A warp is just a hardware implementation thing specific to nvidia.  But afaik, all threads in a warp are executing the same code at the same time: so they have to be part of the same opencl workgroup for it to make any sense.

i.e. i believe there is a 1:N mapping of opencl workgroup to nvidia warp.

Although there might be room for further improvement, at least I can see that the GPU is actually faster than the CPU, so I’m satisfied for now. The only thing I can’t quite guess is why the same program runs about 8 times slower than the CPU on my old GeForce 8200, even when I optimize the execution parameters. I guess that is because it’s an onboard GPU and global memory accesses are even slower than on a GPU with dedicated memory. The same is true when I execute the CL program on my CPU device, but it might just be too massively multithreaded for a CPU, I haven’t tested this enough yet.

Anyway, thanks for your help!

Well there’s a large variation in performance of gpu cards, so it can’t speed them up.

And to get that performance you need to access memory properly - i.e. coalesced.

I have a follow up question to this.  In my GPU there are 384 cores, 8 compute units (streaming multiprocessors), so there 384/8 = 48 streaming processors on each compute unit.  Given that NVidia warp size is 32, which means 32 threads execute in step, doesn’t that mean 16 SPs are not doing anything on each cycle? That doesn’t seem to make sense to me.  Can someone help to clarify?

Thanks,
J

Powered by Discourse, best viewed with JavaScript enabled


How to optimize tail effect?

Hi Experts,

I have been optimizing a kernel function for several days now, and I believe there is no more room for optimization in terms of mathematics and algorithms. It’s time to focus on CUDA programming for further optimization.

According to Nsight Compute, the tail effect is the biggest bottleneck.

My device is the AGX Orin, with 16 SMs. Currently, gridDim = dim3(121, 3), blockDim = 128 (4 warps).

I understand that the tail effect is caused by an imbalance in the workload of the last wave, and I may need to fill the remaining blocks in the last wave.

I have also referred to CUDA Pro Tip: Minimize the Tail Effect | NVIDIA Technical Blog

I would like to know if my understanding is correct, and what would be the best approach for optimization.

Thanks!

圖片1703×102 17.1 KB

Tail Effect: Est. Speedup: 50%
A wave of thread blocks is defined as the maximum number of blocks that can be executed in parallel on the target GPU. The number of blocks in a wave depends on the number of multiprocessors and the theoretical occupancy of the kernel. This kernel launch results in 1 full waves and a partial wave of 171 thread blocks. Under the assumption of a uniform execution duration of all thread blocks, the partial wave may account for up to 50.0% of the total kernel runtime with a lower occupancy of 32.6%. Try launching a grid with no partial wave. The overall impact of this tail effect also lessens with the number of full waves executed for a grid.

3 * 121 = 363 = 192 + 171

You have two waves, one with 192 blocks, one with 171 blocks. That is quite balanced.

Not sure, where the 32.6% is coming from.

I would doubt that you achieve a speed-up of 50% without getting more work to the GPU with one invocation (e.g. two iterative kernel launches put into one kernel launch).

Thank you for your reply!

This is a single kernel, so there’s no issue with iterative or consecutive launches.

In fact, I often doubt whether the optimization suggestions provided by Nsight Compute can actually be achieved…

This is a single kernel, so there’s no issue with iterative or consecutive launches.

I was more thinking of combining with other launches to make the invocation more efficient.
With two waves the amount of tail effect is higher than with more waves. Or with parallel kernel launches on other streams, etc.

If you want to explore it further, the usual suggestion (I think) would be to rewrite the kernel as a grid-stride loop, and choose a grid size to exactly fill your GPU.  That will reduce your kernel to a single wave.  The 50% number is based on the idea that that will be the most efficient launch (with respect to occupancy and the tail effect), and “could” result in a kernel duration equivalent to a single wave of your current realization.

That could be viewed as an “upper bound” or best case outcome.  So  if you wish, interpret the nsight compute suggestions that way.  The percentage speedup is “the most you could possibly get from this change, if all other conditions were perfect for the effect.”

Since nsight compute can’t (currently) do the level of analysis needed to accurately predict the actual speedup from a complex refactoring, then it gives the output in that sense.

Thanks for the hints!

After switching to grid-stride and reusing threads, the duration dropped to 18 microseconds, showing 25% improvement. Additionally, the Tail Effect: Est. Speedup: 50% suggestion has disappeared from the optimization opportunities section.

I’m curious whether the 50% improvement mentioned in the NCU report can really be achieved.

圖片1544×305 85.1 KB

Hi,
with just two original two waves, one should in general take special care, when using grid-stride loops, how the work is distributed on different SMs and SM Partitions to avoid that one SM Partition has finished early and others have several more blocks to do (which is like an implicit tail effect). That is especially relevant, if one SM does more than 4 warps (i.e. one SM Partition has more than 1 warp).
In your case, each SM Partition has one warp, so it is less relevant.

I want to mention briefly that the previous results had some bugs.

The final results, which used grid-stride loops, did not show significant gains. I might share the code later.

I wouldn’t really expect much gains from the grid-stride refactoring by itself, for a case like this (2 waves).  As curefab points out, you may not be changing the work breakdown structure very much, and even though you have eliminated the “wave effect”, you probably haven’t done much to actually make the imbalance go away.  Naturally this would depend to some degree on how much work you are actually doing per element or per thread.  If the work per element or per thread is “large” then the refactoring is likely to make little difference.

To see something approaching a large difference in performance (like 50%) you would need a situation where the work per thread or per element was almost zero, such that the overhead of work scheduling (e.g. deposition of blocks, traversal of loops, etc.) was dominating your performance.  Such characteristics are not a hallmark of good CUDA code anyway, although people sometimes wrestle with those cases as well.

Related topics

Powered by Discourse, best viewed with JavaScript enabled


vLLM Office Hours: Get the latest updates, connect with committers, and up-level your vLLM skills. Join us!

Products

Discover faster ways to inference your ML model.

Products

nm-vllm

Enterprise inference server for LLMs on GPUs.

Neural Magic Compress

Developer subscription for enterprises aiming to build and deploy efficient GenAI models.

DeepSparse

Sparsity-aware inference server for LLMs, CV and NLP models on CPUs.

Community

Explore essential resources for every ML practitioner.

Community

vLLM Office Hours

Join our bi-weekly vLLM office hours to learn, ask, and give feedback.

GitHub

Look under the hood and contribute to our open-source code.

SparseZoo

Get started faster with our open-source model repository.

Hugging Face

Deliver fast inference with our pre-optimized, open-source LLMs.

Docs

Access the tutorials, guides, examples, and more.

Blog

Resources

Peruse our research. Ask a question.

Resources

Research Papers

Learn more about the magic behind Neural Magic.

Support

Get the answers you need.

Company

Get to know us better.

Company

About Us

Who's Neural Magic?

Our Technology

How does it work?

Careers

Interested in joining our team?

Contact

Have a question for us?

Let's Connect

Let's Connect

Introducing Machete, a Mixed-Input GEMM Kernel Optimized for NVIDIA Hopper GPUs

Oct 14, 2024

Author(s)

Mixed-input quantization is a technique that processes weights and activations at different precisions in neural networks. The most common implementation is w4a16 quantization (e.g., GPTQ or AWQ), which uses 4-bit quantized weights and 16-bit activations (float16 or bfloat16). This approach primarily aims to reduce GPU memory requirements for model execution.

In most Large Language Model (LLM) workloads, model weights consume the majority of GPU memory. By quantizing weights from float16 to 4-bit, a remarkable ~4x reduction in memory needed for weight storage can be achieved. Additionally, smaller weights can lead to speedups when the linear layer is memory-bound (i.e., limited by weight loading), which occurs when the batch or sequence length is small, resulting in activations being much smaller than the weights.

LLM inference involves a mix of both compute-bound and memory-bound iterations. On modern NVIDIA Hopper GPUs, current state-of-the-art mixed-input linear kernels struggle in compute-bound scenarios (as illustrated in Figure 1).

We are excited to announce Machete, Neural Magic's latest advancement in mixed-input quantization performance. This kernel is the spiritual successor to the Marlin kernels created by Elias Frantar and integrated into vLLM by Neural Magic. While Marlin was specifically designed for Ampere generation GPUs and struggles on Hopper GPUs (namely H100), Machete was built on top of the work highlighted in CUTLASS 3.5.1 (see example #55 as our initial starting point). This allows it to efficiently target Hopper and beyond, performing well in both compute and memory-bound regimes. It optimizes the on-the-fly upconversion of weights required in mixed-input scenarios, and hides this latency by overlapping it with compute and data-movement.

Machete is now available in vLLM 0.6.2+ as a backend for w4a16 and w8a16 compressed-tensors models, for GPTQ models, and more to come. With Machete, you can now serve Llama 3.1 70B on a single H100 GPU with up to 5 user requests per second while maintaining a median time to first token (TTFT) of <250ms and a median time per output token (TPOT) of <100ms (using chunked prefill on the ShareGPT dataset).

With Machete you can now also hit those same serving targets for Llama 3.1 405B using 4 H100 GPUs with up to 3 user requests per second.

NOTE: Our use of the term "mixed-input" rather than "mixed-precision" is deliberate, as it more accurately describes the specific case we're addressing. The term "mixed-precision" has traditionally been used to describe a broader range of cases, namely including the case where activations and weights share the same type but are accumulated into a different type (i.e. w8a8).

Optimizing Mixed-Input Linear Operations: Weight Pre-Shuffling

While Neural Magic's previous blog post on Marlin covered many optimizations for mixed-input linear operations, this article focuses on an important previously undiscussed optimization used by both Marlin and Machete: weight pre-shuffling. To understand the benefits of weight pre-shuffling, we first need to examine how data is fed into the tensor cores in NVIDIA GPUs.

When performing matrix-input multiplication on a GPU, the process begins by loading data for a small subproblem from global memory to an SM's local shared memory. This data is then transferred to threads and passed to the tensor cores. Each thread is responsible for loading and holding a specific piece of data in its registers. The layout of this data in registers follows a fixed, complex pattern that varies depending on the instructions used (such as mma or wgmma).

In PyTorch, weights are typically stored in row-major or column-major format, which doesn't align with the intricate layout required by tensor cores. This mismatch creates a challenge: while we can load data into shared memory in row-major format, we must shuffle it when loading into registers to match the tensor core requirements.

For the purposes of illustration in the following animations, we're using a fictitious GPU that has 8 threads per warp and tensor cores that operate on 8x8 chunks of the weight matrix. While simplified, this closely matches the types of layouts used by NVIDIA tensor cores, albeit scaled down. In these diagrams, we only show the weight matrix (and not activations) as loading and up-converting the weight matrix is the main challenge in mixed-input linear layers.

For standard data types (16, and 32-bit), NVIDIA provides an efficient ldmatrix instruction to perform this shuffling in hardware; i.e. ensuring that the right data gets shuffled to the right thread.

However, this instruction isn't available for 4-bit types. When working with 4-bit elements and using float16 or bfloat16 as the compute type, we need to load 4-bit elements to match the thread layout for a 16-bit type. Without a 4-bit ldmatrix instruction, we would naively need to resort to performing four 8-bit shared memory loads per tensor core operation. In this case the data shuffling is being handled by software using multiple shared memory loads. These additional shared memory loads are detrimental to performance, as they add latency and the use of only 8-bit loads restricts the shared-memory to registers bandwidth.

To overcome this limitation, we can reorder the data ahead of time. By doing so, we can perform a single 32-bit load from shared memory instead of four 8-bit loads. This approach is much more efficient in terms of shared memory bandwidth and latency, ensuring we don't get bottlenecked waiting for shared memory. Importantly, all global memory reordering is done in advance, so it doesn't impact inference time. This pre-shuffling and its effect on how the data gets loaded into memory can be seen in Animation 3.

We can push this optimization even further. By interleaving data for four tensor operations together (e.g., interleaving four 8x8 tiles in the visualization), we can perform 128-bit loads—the widest shared load instruction currently available on CUDA devices.

After loading the weight parameters into the correct registers, they must be upconverted to 16-bit. Animation 5 demonstrates this process, highlighting how the interleaving of tiles can simplify upconversion and save instructions. By interleaving tiles in global memory, the data is arranged so that, once in registers, multiple nibbles can be efficiently extracted and up-converted in parallel. This is achieved by shifting the nibbles into the lower four bits of their destination registers using simple bit shifts and masking operations and then expanding in-place. If you're curious about these interleaved upconverts, you can find them here.

What's New in Machete vs. Marlin?

These types of repackaging techniques have already been used in previous mixed-input kernels (namely Marlin and AWQ), so what does Machete do differently? The motivation for developing Machete mainly stems from the poor performance of current mixed-input linear kernels on the NVIDIA Hopper architecture when it comes to larger, more compute bound matrix-multiplications, as can be seen in Figure 1.

New Tensor Core Instructions (wgmma)

For Marlin, the poor performance on Hopper architecture primarily stems from the use of outdated 'mma' tensor core operations. To achieve peak FLOPs on NVIDIA Hopper, the new 'wgmma' instructions must be utilized. Using only 'mma' instructions results in a loss of approximately 37% of peak compute throughput [1, 2].

Marlin's weight pre-shuffling, being hand-derived and implemented, makes it challenging to easily adapt to the new 'wgmma' layouts. Machete circumvents this issue by employing CUTLASS CUTE layout algebra to construct a description of the repacked layout for a full weight matrix using instruction layout definitions available in CUTLASS. This approach should, in theory, facilitate easier adaptation to any future instructions as well as different types (w4a8 is already in progress).

A key challenge with 'wgmma' is that for a matrix multiplication C = AB, only A and C can be sourced from registers, while B must be sourced from shared memory. Since we upconvert the 4-bit bit weights to 16-bit floating point values in registers, we can avoid restoring them into shared memory by computing Y^T = W^T * X^T instead of Y = XW. This ensures the weights are ‘A’ (left side input-operand) with respect to the ‘wgmma’ instructions, allowing them to be sourced directly from registers. CUTLASS enables us to more easily compute the transpose problem by simply manipulating layouts.

Tensor Memory Accelerator (TMA)

The Tensor Memory Accelerator (TMA) represents a significant advancement in NVIDIA Hopper GPUs' memory handling capabilities. This new hardware feature is designed to asynchronously copy blocks of multidimensional data, known as subtensors, from global memory to shared memory. The introduction of TMA brings several important benefits to the table.

Primarily, TMA reduces register pressure by offloading data movement operations, thereby freeing up CUDA cores for other computational tasks. It also simplifies address calculations by handling these complex operations in hardware. Furthermore, TMA's ability to operate independently of compute operations allows for better overlap between memory transfers and computations. Machete takes advantage of this new hardware feature by leveraging CUTLASS's existing TMA infrastructure.

Warp-specialization

Warp-specialization, introduced in CUTLASS 3.0, divides warps into data movement (producer) and computation (consumer) roles. This technique aims to better overlap data movement and computation, improving memory and tensor core latency hiding. Machete incorporates this approach by leveraging existing infrastructure in CUTLASS. For a more detailed explanation of warp-specialization in CUTLASS, refer to this COLFAX Research blog.

Machete Performance

With all of the above optimizations in place, we can see that Machete outperforms the other mixed input linear kernels for batch size / prefill seq. len 128+. At batch sizes of 128 and above, the performance is competitive with FP16, meaning there is no longer a trade-off between prefill performance or high-batch size performance and improved low-batch and decode performance.

In Figure 6 we can see end-to-end serving performance of these kernels on a 4bit Llama 3.1 70B on a single H100. In the higher user requests rates (3+ req/s), we see a geomean speedup of 29% for input token throughput and 32% for output token throughput.

In Figure 7 we can see end-to-end serving performance of these kernels on a 4bit Llama 3.1 405b on 4 H100s. In the higher user requests rates (3+ req/s),we see a geomean speedup of 42% for both input token and output token throughput.

Future Work

As we continue to develop and refine Machete, we have several exciting areas of focus for future improvements:

These initiatives underscore our commitment to pushing the boundaries of mixed-input quantization performance. By addressing these areas, we aim to make Machete an even more powerful and flexible tool for efficient LLM inference on NVIDIA Hopper GPUs and beyond. We're excited about the potential impact of these improvements and look forward to sharing updates as we progress. Subscribe to our blog, follow us on X, and join our bi-weekly vLLM office hours to stay tuned for more exciting AI developments.

About Neural Magic

Neural Magic is advancing the performance of AI inference by optimizing large language models (LLMs) for efficient and scalable deployments. As a leading contributor to the open-source vLLM project, we develop and implement key techniques like sparse architectures, mixed-precision quantization, and performance optimizations to enhance inference speed, reduce memory footprint, and maintain model accuracy. Neural Magic is also a member of NVIDIA Inception, a program designed to nurture startups, and is thankful to the CUTLASS team for their valuable work. Our goal is to empower developers to build and deploy high-performance LLMs across different hardware configurations without compromise. To learn more, visit neuralmagic.com or check out our GitHub to accelerate your AI workloads today.

Author(s)

Spread the Word

Stay Up to Date

SUBMIT

Join the Conversation

slack

Continue Reading

Recent Blogs

Open Source

Feb 27, 2025

Quantized DeepSeek-R1 Models: Deployment-Ready Reasoning Models

Newsletter

Feb 24, 2025

Friends of vLLM: February 2025

Open Source

Feb 20, 2025

Driving Enhanced Support for Multimodal LLMs With vLLM V1

view all blogs

SUBSCRIBE

Subscribe to Neural Magic events & news

Community

Blog

Let's Connect

Contact Us

Company Policies

BACK TO TOP

© 2024 Neuralmagic, Inc.

Neuralmagic, Inc.    55 Davis Sq STE 3    Somerville, MA 02144    United States

BACK TO TOP


Optimizing memory-bound kernel (memory dependency around 95% in NVVP)

I have a piece of code that, according to Nvidia Visual Profiler, is memory bound and so far I haven’t managed to improve it further after passing some arguments as constants.
If you copy/paste and compile the following code, NVVP shows that both kernels are memory bound, have around 89% occupancy even though the kernel configurations should fully saturate the device, and the SMs from 1 to 7 are around 88-90% utilization while the other ones are closer to 100%.
Error checking was omitted for easier reading, but cuda-memcheck reports no errors for any array length I use.

#include <iostream>

__global__ void init_array(float *array, size_t len)
    {
    for(size_t idx = blockDim.x * blockIdx.x + threadIdx.x; idx < len; idx += gridDim.x * blockDim.x)
        array[idx] = idx;
    }

__global__ void transform_array(float *in, float *out, const float scale_factor, size_t len)
    {
    for(size_t idx = blockDim.x * blockIdx.x + threadIdx.x; idx < len; idx += gridDim.x * blockDim.x)
        out[idx] = in[idx] * scale_factor;
    }

int main(void)
    {
    float *array_in, *array_out;
    size_t length = 100000000;
    const unsigned short block_Size = 256, grid_Size = 200;
    const float factor = 0.5;

    // Allocate and initialize memory
    cudaMallocManaged(&array_in, length * sizeof(float));
    cudaMallocManaged(&array_out, length * sizeof(float));
    cudaMemset(array_in, 0, length * sizeof(float));
    cudaMemset(array_out, 0, length * sizeof(float));
    cudaDeviceSynchronize();

    // Fill the input array
    init_array <<< grid_Size, block_Size >>> (array_in, length);
    cudaDeviceSynchronize();

    // Transform input and write to output array
    transform_array <<< grid_Size, block_Size >>> (array_in, array_out, factor, length);
    cudaDeviceSynchronize();

    cudaFree(array_in);
    cudaFree(array_out);
    return 0;
    }

The first kernel just initializes the input array with some numbers using a strided loop, and second kernel saves the multiplication between the input element and some scaling factor (which I calculate with other functions, but here it is just an arbitrary value) to the output array, again using the same strided loop. Essentially doing a lot of work in global memory.

How do you normally get rid of/alleviate this bottleneck?

You won’t eliminate the memory bottleneck for a memory bound code.  The operations you are doing here are so trivial they are going to be memory bound.

There is likely very little you can do to make them run substantially faster.  At this point, if you want to improve things, you are in the realm of what I call “ninja methods”.  Things like tuning kernel size (e.g. number of blocks - easily doable with your grid-stride loop method) for the number of SMs in your device to minimize the tail effect, attempting to see if larger vector loads will improve things (slightly), etc.

Ninja methods are referred to here:

[url]http://on-demand.gputechconf.com/gtc/2012/presentations/S0514-GTC2012-GPU-Performance-Analysis.pdf[/url]

These methods in my experience don’t usually provide more than a few percent improvement.

At a higher level of abstraction, programmers who have multiple operations like this to do will sometimes seek to fuse operations.  This means combining multiple kernel calls to do more work in a single kernel call.  The objective is to do as much work as possible per load and store operation in global memory.  Your two operations could be trivially fused into a single kernel, for example.  Fusing to reduce kernel calls also saves the overhead of additional kernel calls - another ninja topic (usually).

In any event, these trivial memory-bound kernels are “fully optimized” when the kernel runs at the rate of memory bandwidth.  For example, determine the total number of loads and stores done by a kernel, in bytes, and divide by the kernel execution time, in seconds.  This bytes/sec number is then compared to a proxy measurement of peak achievable bandwidth (e.g. such as the device-to-device memory bandwidth reported by bandwidthTest sample code).  When your kernel is running at that rate, it probably cannot be optimized further.  You are done, excepting higher-level “meta” work like algorithm redesign or fusing of operations/kernels.

Thanks for these clarifications, txbob. So I think it is just what it is. And the Titan V with that ridiculous memory bus is probably laughing at it.
But while I was reading this document and trying some of the ninja techniques, like increasing the grid size to raise the occupancy (with 200 it had 89%, with 1000 it goes to 98%) and shaving some milliseconds here and there, I found by accident, clicking the wrong kernel to profile, that the array reduction we worked some weeks ago actually has some branch divergence, doesn’t it?

It is exactly the last lines:

if (tid == 0)
    array_out[blockIdx.x] = sdata[0];

Only a few threads will execute it, so I don’t think it is all that harmful, yes? NVVP shows an increase in divergence as the grid size increases.
There is probably an expression in English like, you aim at something but hit something else…

Many, many kernels will have some divergence.  your grid-stride loops, for example, are prone to some small divergence as well.  These sorts of things are usually insignificant, from a performance perspective.

Related topics

Powered by Discourse, best viewed with JavaScript enabled


Keeping data between kernels calls

Hi,

I have a quite long kernel that I want to optimize. My idea is to split this single kernel into multiple smaller ones that can be better optimized. The problem I see is now that the result of one kernel is the input data of the next. So far the only solution I found to keep the data between different kernel calls is the use of global memory. If I unterstand correctly the lifetime of shared memory does not allow the use for different kernels - so that is unfortunately not an option.

Is there any other way to keep the data?

Thanks.

My idea is to split this single kernel into multiple smaller ones that can be better optimized.

Better optimized in which sense? What factors will be driving the performance improvements that are being envisioned?

So far the only solution I found to keep the data between different kernel calls is the use of global memory.

Keeping data resident on the GPU and minimizing data transfers between host and device is the correct approach.

Hi,

thanks for your fast reply. Some part of the kernel can make use of a whole warp to solve a single problem, other parts may use just a single thread for a fast calculation. This is alternating through the kernel. So I want to separate the parts that can make use of more than a warp to solve a problem from the other that needs just a single thread.

Hope I could make my point clear?

I would hope that the kernel launch comprises more than one warp, because that would still be very inefficient. As for the single-threaded portions: I can see how this may be necessary if there is simply no inherent parallelism available, but I am not sure what kind of situation “just a single thread for a fast calculation” refers to.

Generally speaking, the kind of situation you describe, with varying degrees of parallelism available throughout a kernel, is not uncommon and may well be the highest-performing option. A thorough design process would prototype both this design approach as well as the split-kernel approach to assess which one is faster for this particular use case.

The other aspect that is likely worth investigating is whether there are algorithmic changes that could be used to increase the amount of available parallelism, so as to avoid single-threaded computation. Alternatively or additionally, since single-threaded execution is ideally suited to the CPU, one might alternatively look into how these portions of the computation could be moved to the host, maybe in the form of pre-computation.

A thorough search of the literature might turn up interesting ideas. How to best map work to execution resources on massively parallel computational platforms is still an area of active research. Sometimes a switch from traditional data structures can be the key to better parallelization. By now the applicability of GPUs to just about any kind of computational task has been looked into, so a literature search is highly likely to yield some useful information.

The process I like to use for best results is to first spend some quality time thinking up a design and prototyping it (usually with variants) by myself. I put serious effort into this. In a second stage I then search the literature extensively, going back multiple decades, looking for anything even remotely applicable.

Having made a serious first effort by myself usually provides me with enough insight that I can make a reasonable assessment whether some idea from the literature is (1) roughly along my own line(s) of though, (2) inferior to my own ideas (3) superior (at least in parts) to my best efforts. It is fairly rare (but happens), that I realize that an approach from group (3) is so vastly superior to my own ideas that I adopt it wholesale. Most of the time my final design is the synthesis of my original ideas and ideas from the literature: the “stand on the shoulder of giants” approach.

Thanks again for your detailed answer. Yes, I agree that prototype is one way to go (already started).

Just to clarify my approach since I think I didn’t explain very well what I try to do.

With ‘single threaded’ I do not mean that I start a kernel with a single grid and 32 threads. What I mean is that each thread gives me a single solution. I’m also happy with the performance of these parts.

In other parts of the kernel I can use 4 thread to improve performance, also giving me a single solution, and in other parts I can make use of a complete warp.

If all of these parts work in a single kernel I need to configure the kernel in that way that the part that needs most of the threads determine how many solution my kernel can calculate.

Sample:

If I start now a kernel like this:
grid <64, 1, 1> and block <128, 1, 1>

I use 64 * 128 threads that gave me 256 solutions since the last part needs 32 threads for a single result. And on the other side I have most of the GPU doing nothing for part 1 and 2.

My idea for splitting the kernel is now to run part 1 in its own kernel with a good occupancy and store all results in memory. The run part 2, reading the results from part 1 and continue calculation also in configuration that its kernel make good use of the gpu - then finnaly run the last part - again with its own kernel configuration.

I hope to get better performance because part 1 and 2 can use their own configuration resulting in a better occupancy. I can also use the unused MC for something else in a different stream (if the used resources will allow this). But this something for a later development if splitting the kernel gives the expected results.

Thanks for your help and insights.

Hi TrailingStop,
Variant 1:
it can be done by e.g. using a single kernel and have each warp calculate 32 solutions. In your example: For the first part of the kernel, each thread of the warp calculates one solution; for the second part, you have a for loop from 0 to 3 and within the loop body 4 threads each cooperate for 8 independent calculations concurrently times 4 loop iterations; for the third part you have a for loop from 0 to 31 with only one solution calculated for each iteration.

The data can be exchanged by warp shuffle or by shared memory.

Variant 2:
A different approach would be to have whole warps doing nothing instead of some threads of a warp. Keeping warps idle (e.g. by blocking for a _syncthreads()) is free, because the blocked warps are not scheduled and do not use up compute resources, except that the effective occupancy lowers.

Your example would be a (too) extreme case. But basically it would go: Start 32x32 thread blocks. For part 1 one warp is active; for part 2 fours warps are active; for part 3 thirty-two warps are active.

The data would be exchanged by shared memory.

Generally:
The thread indices of threads within a block can have different meanings throughout your kernel. You are not bound to map them 1:1 to your problem space indices. Just see the threads as numbered resources. Between each _syncwarp() (or even _syncthreads()) you can redefine, how the work is distributed onto threads, and you can add any for loop within the kernel on top of it.

Hi Curefab,

thanks a lot for this information. I will give them a try and check which one works best in my case.

Thanks!

Related topics

Powered by Discourse, best viewed with JavaScript enabled


Why performance is worse with CUBLAS- than with kernel-function

System:
CPU: Intel Core i5-4570
MSVS Community 2017 v15.9.7
Platform Toolset: Visual Studio 2017 (v141)
Build: Release x64
GPU: GeForce GT 640 (c.c. 3.0)
CUDA Compilation tools R10.1, V10.1.105
CUDA Driver Ver.: 10.1

I am using CUDA for last couple of months. Goal of my research is to develop performance optimized 2D DCT transform kernel function. Optimization targets short processing time. Since transform is used for video processing batches of data are processed. Transform can be described with mathematical equation C = A * B * AT where A and AT are predefined matrices. All matrices are of size 32 x 32.

Own kernel function was developed at first and to check potential improvement variant with CUBLAS function was developed as well. Function cublasgemmBatched()was used for this purpose. It was used twice for two multiplications from math eq. Batch size is 12960. Results were compared at the end for both variants. I expected that transform variant with CUBLAS will be faster but processing time with kernel function is almost 10x faster. How to explain this ?

Any idea where to search for an answer ?

Should I try with strided batched matrix multiplication cublasgemmStridedBatched() since I operate with square matrices only ? Or there is another library which should be used to outperform the kernel function ?

I know I ask to many questions :) but any suggestion is welcomed.

There was problem that I didn’t have warmup call. Its absence was of significant importance for CUBLAS variant since after warmup was added for both, CUBLAS and own kernel, processing time for CUBLAS variant was 1,38x shorter.

In next iteration I moved to cublasgemmStridedBatched(). There CUBLAS showed even better performance, it was 1,73x faster.

Using NVIDIA visual profiler I identified that low shared efficiency (50%) in my own kernel had improvement potential. After rewriting my kernel for vectorized memory access with float2 data type shared efficiency increased to about 98% and my kernel was 1,3 faster than CUBLAS.

Now, I don’t know why CUBLAS exhibits such low performance. In the profiler I see that block size for CUBLAS kernel is relatively small (8x8) what decreases number of possible active warps per SM and decreases the occupancy (30%). Global store efficiency is also low, 25%. Is there an issue that CUBLAS GEMM is optimized for specific matrix sizes and my application with 32x32 matrices is not in that group ? Probably I will move to Tesla K40 GPU and see does GPU architecture makes the difference.

Generally speaking, GEMM maps to dozens of different kernels, optimized for different GPU architectures, matrix sizes, transpose modes, matrix aspect ratios etc. For a given GEMM invocation a heuristic picks the most appropriate kernel(s). The heuristic may not always pick the optimal kernel, or none of the available kernels may be the perfect fit for a particular call to GEMM.

As I recall, batched GEMMs in particular were introduced primarily to deal with very small matrices, as some applications need to handle tons of matrices of size 3x3, 4x4, or thereabouts. Matrices of size 32x32 may be close to the upper limit of what batched GEMMs were targetted to handle; check the documentation.

With regard to the sub-optimal performance observed, consider filing an enhancement request with NVIDIA. You can file one by using the bug reporting form and prefixing the synopsis with “RFE:” to mark it as an enhancement request.

Realistically, given the age of the Kepler architecture, it is unlikely that improvements will be made for compute capability 3.x, but equivalent issues may affect newer architectures. The primary targets for performance improvements in libraries are the latest GPU architectures (at this time: Turing and Volta) although some amount of back-porting of such improvements to older architectures may occur.

Related topics

Powered by Discourse, best viewed with JavaScript enabled


For context: this WebGPU version achieves ~17% of peak theoretical performance of M2. With CUDA (i.e. CuBLAS), you can reach ~75% of peak performance for same matrix config (without tensor core).

Not on the same computer, CUDA doesn’t run on the integrated GPU of the Apple M2 Pro.

That single lib call must have used the AMX accelerator, which is separate from the cores and shared by a group of cores.So that AMX accelerator performance may be greater than of all CPU cores together. AFAIK, some Apple CPUs have one AMX accelerator for the big cores and another AMX accelerator for the smaller cores, but in any case there is no chance to hope that if you have obtained 1 TFLOP/s when running the program on 1 core you will get much more when running it on multiple cores, because all cores of the same type will use the same shared accelerator.

So that AMX accelerator performance may be greater than of all CPU cores together. AFAIK, some Apple CPUs have one AMX accelerator for the big cores and another AMX accelerator for the smaller cores, but in any case there is no chance to hope that if you have obtained 1 TFLOP/s when running the program on 1 core you will get much more when running it on multiple cores, because all cores of the same type will use the same shared accelerator.

nVidia tensor cores support int8, couple versions of FP16 (BF16 and the standard IEEE one) and FP19 which they call TensorFloat-32. I think Intel AMX only supports int8 and BF16.None of them supports FP32 let alone FP64 input numbers, which makes them completely useless for traditional GEMM stuff.

None of them supports FP32 let alone FP64 input numbers, which makes them completely useless for traditional GEMM stuff.

The Apple’s version is indeed interesting. I wonder why haven’t Apple exposed it to programmers, or implemented a BLAS library on top of that thing?

Using the Accelerate framework (which includes Apple's BLAS) is the only supported way for programmers to access the AMX. Reverse engineering the instruction set to access it directly is discouraged, because it's not a documented stable interface.

Now that they’re using “standard” SME, it shouldn’t be a problem to write SME assembly opcodes directly, although I suspect Apple themselves is still probably sparse on the documentation. I’m not aware if there’s any way to use intrinsics or something slightly higher level than inline-ASM, but lower level than the Accelerate framework.

It sounded like you claimed that using only one core you already reach 1 TFLOP/s, implying that you could reach more than that by using more cores, which is false.Now you have clarified that you actually claim that it is good that when using a single core you can reach the maximum throughput of the shared matrix operation accelerator.This is correct, but there is no essential difference between this and a Zen 5 CPU that reaches this throughput by using only half of the cores, while having the other half of the cores free to do any other tasks.

Now you have clarified that you actually claim that it is good that when using a single core you can reach the maximum throughput of the shared matrix operation accelerator.This is correct, but there is no essential difference between this and a Zen 5 CPU that reaches this throughput by using only half of the cores, while having the other half of the cores free to do any other tasks.

This is correct, but there is no essential difference between this and a Zen 5 CPU that reaches this throughput by using only half of the cores, while having the other half of the cores free to do any other tasks.

(Also, that’s a M2 number, since that’s what OP was talking about. Someone will presumably post M4 benchmarks for BLAS sometime soon, if they haven’t already.)

This is way way lower than what you claim M2 Pro is capable of and since I'm comparing it against the state-of-the-art datacenter CPU I'm curious how did you get to this number?M2 Pro core runs at much lower frequency, what it seems to be around ~3.4GHz. And I couldn't find any information about SVE vector widths supported nor number of FMAs.

M2 Pro core runs at much lower frequency, what it seems to be around ~3.4GHz. And I couldn't find any information about SVE vector widths supported nor number of FMAs.

In my desktop computer, I have Ryzen 7 8700G CPU, which has 8 Zen 4 cores, 4.2 GHz base frequency, 65W TDP. Theoretically, when doing FP32 FMA, each CPU core can do 32 FLOP/cycle. At the base frequency, this translates into 134 GFlops per core. You gonna need all 8 cores to achieve 1 theoretical TFlops.BTW, integrated GPU inside the same 8700G processor can theoretically do 8.2 TFlops FP32.

BTW, integrated GPU inside the same 8700G processor can theoretically do 8.2 TFlops FP32.

Isn't it that zen4 doesn't have "native" support for AVX-512 but "mimics" it through 2x 256-bit FMA units?Because of this, a single AVX-512 instruction will occupy both FMA units and therefore I think that the theoretical limit for a single zen4 core should be half of the 134 GFLOPS number?

Because of this, a single AVX-512 instruction will occupy both FMA units and therefore I think that the theoretical limit for a single zen4 core should be half of the 134 GFLOPS number?

According to uops.info, Zen 4 cores can do two 8-wide FMA instructions per cycle, or one 16-wide FMA per cycle. See VFMADD132PS (YMM, YMM, YMM) and VFMADD132PS (ZMM, ZMM, ZMM) respectively, the throughput column is labelled TP. That’s where 32 FLOP/cycle number comes from.> doesn't have "native" support for AVX-512 but "mimics" it through 2x 256-bit FMA unitsThat’s correct, AVX512 doesn’t deliver more FLOPs on that CPU. The throughput of 32-byte FMA and 64-byte FMA is the same, 32 FLOP/cycle for FP32 numbers.

> doesn't have "native" support for AVX-512 but "mimics" it through 2x 256-bit FMA unitsThat’s correct, AVX512 doesn’t deliver more FLOPs on that CPU. The throughput of 32-byte FMA and 64-byte FMA is the same, 32 FLOP/cycle for FP32 numbers.

That’s correct, AVX512 doesn’t deliver more FLOPs on that CPU. The throughput of 32-byte FMA and 64-byte FMA is the same, 32 FLOP/cycle for FP32 numbers.

Right. This is where the discrepancy comes from. I counted FMA as a single FLOP.

For example, I have GeForce 4070 Ti Super in my desktop. The chip has 8448 execution units; nVidia calls them CUDA cores but I don’t like the name, the correct number is 66 cores where each core can do 4 wavefronts of 32 threads each.
Anyway, these EUs can do one FP32 FMA each cycle, and the boost clock frequency is 2.61 GHz.
Multiplying these two numbers results in 22.04928E+12 cycles*EU/second, and nVidia reports 44E+12 FLOPs peak FP32 performance of the GPU.

That can lead you to some pretty counter-intuitive optimizations because it's often faster to do more compute work if it means you touch less memory in the process.

It is not specific to the GPUs: this kind of optimizations are pretty common on CPU too where latency kills you and 200 cycles spent wasted on doing compute can actually be faster than a single cache miss trying to fetch data. This is pretty common for many SIMD algorithms actually.Memory is currently lagging behind compute on almost every type of modern hardware, and it will very likely become worst, not better.

Memory is currently lagging behind compute on almost every type of modern hardware, and it will very likely become worst, not better.

As for handcoded assembly, do you believe that it would be financially sound to hand code and maintain thousands of kernels that way, even if you believed that they would be faster?

Why not? We do so for cryptographic primitives and video codecs. And why are you talking about “thousands of kernels”? AI programs only need a small amount of different kernels so it doesn't sound intractable.

That is not the case. What appears like a simple matmul operation actually requires these libraries to select which specific kernel out of the many internally available to execute.If you are curious to learn more, NVidia open sourced a library called Cutlass some years ago. And remember that is only what they are willing to open source.

If you are curious to learn more, NVidia open sourced a library called Cutlass some years ago. And remember that is only what they are willing to open source.

How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance
(https://siboehm.com/articles/22/CUDA-MMM)(It's CUDA-specific, so there may be aspects that can't yet be ported to WGPU)

(It's CUDA-specific, so there may be aspects that can't yet be ported to WGPU)

there are a few things that i wasn't able to figure out how to get access to/i wasn't sure if they were possible. for example, a lot of Simon's article takes advantage of the warp scheduler and warp tiling.i had a hard time finding information on if that's even possible with my M2/metal and the general memory access patterns. it seems like CUDA does have better documentation in this regard

i had a hard time finding information on if that's even possible with my M2/metal and the general memory access patterns. it seems like CUDA does have better documentation in this regard

The datasheet for the H100 SXM seems to indicate that it can only do ~1000 TFLOP/s peak.

I am not an expert in LLM but I don't think you can end up having a significant amount of zeroed weights (~50%) in a converged network so I think it is safe to say that the theoretical throughput for 99% of cases is really ~800 TFLOPS and not ~1600 TFLOPS as advertised.

Also does quantized matmuls.

As you see, I have implemented 32×32 tiling, using thread groups of 32×8 threads, two groupshared buffers to load tiles of the input matrices, and I accumulate numbers into local variables, 32 / 8 = 4 accumulators per thread.

I have implemented a profiler on top of D3D11_QUERY_TIMESTAMP and D3D11_QUERY_TIMESTAMP_DISJOINT queries, and tweaked the compute shader to minimize the time reported by these queries for my specific use case.

I have no experience with WebGPU but if you mean group shared memory, I think the support is available. See the demo: https://compute.toys/view/25

i'm excited to try subgroups though: https://developer.chrome.com/blog/new-in-webgpu-128#experime...

it would be cool to see if there's some way to get better access to those lower-level primitives but would be surprisedit does seem like subgroup support are a step in the right direction though!

it does seem like subgroup support are a step in the right direction though!

The smoothness of an iPhone map zoom, on any device.

Any device except an iPhone, until Apple finally gets around to shipping WebGPU in Safari. Any year now...

https://developer.apple.com/documentation/safari-release-not...

"I have found that WebGPU is enabled by default now with iOS 18.2.
Apple has been working in the open on WebGPU. The WebKit source code has their latest WebGPU work in it. What hasn’t been known is their release schedule, but now with 18.2 it’s looking very promising that it will be on by default in that version."

Edit: I just pressed “Reset All to Defaults” under “WebKit Feature Flags” on my device running 18.2 beta, and the switch for WebGPU is on!! <3