Sign up

Sign in

Sign up

Sign in

Mastering CUDA Matrix Multiplication: An Introduction to Shared Memory, Tile Memory Coalescing, and Bank Conflicts

Dhanush

Follow

--

Listen

Share

Introduction

In the world of GPU programming with CUDA, optimizing performance is key. One of the most powerful techniques to achieve this is using shared memory. This blog will walk you through a CUDA program that performs matrix multiplication using shared memory, with a particular focus on understanding tile memory coalescing and bank conflicts. By the end of this post, you’ll have a solid grasp of how shared memory can significantly speed up your computations and how to manage potential pitfalls like bank conflicts.

Understanding the Basics: Shared Memory and Tiling

Shared memory is a special type of memory in CUDA that is much faster than global memory, but with a smaller size, typically a few kilobytes per block. This memory is shared among all threads in a block, making it ideal for optimizing access patterns that involve frequent re-use of data, such as in matrix multiplication.

In matrix multiplication, tiling is a technique where the matrix is divided into smaller submatrices (tiles) that fit into shared memory. These tiles are then multiplied together, which reduces the number of global memory accesses, improving performance.

Let’s dive into the MatrixMultiShared CUDA kernel to see how this works.

The CUDA Kernel: MatrixMultiShared

Below is the CUDA kernel that performs matrix multiplication using shared memory:

__global__ void MatrixMultiShared(float* A, float* B, float* C, int N){    __shared__ float tile_A[TILE_SIZE][TILE_SIZE];    __shared__ float tile_B[TILE_SIZE][TILE_SIZE];int row = threadIdx.y + blockIdx.y * TILE_SIZE;    int col = threadIdx.x + blockIdx.x * TILE_SIZE;    float val = 0.0f;    for(int i = 0; i < (N + TILE_SIZE -1)/ TILE_SIZE; i++){        if(row < N && (i * TILE_SIZE + threadIdx.x) < N){            tile_A[threadIdx.y][threadIdx.x] = A[row * N + i * TILE_SIZE + threadIdx.x];        } else {            tile_A[threadIdx.y][threadIdx.x] = 0.0f;        }        if(col < N && (i * TILE_SIZE + threadIdx.y) < N){            tile_B[threadIdx.y][threadIdx.x] = B[(i * TILE_SIZE + threadIdx.y) * N + col];        } else {            tile_B[threadIdx.y][threadIdx.x] = 0.0f;        }        __syncthreads();        for(int j = 0; j < TILE_SIZE; j++){            val += tile_A[threadIdx.y][j] * tile_B[j][threadIdx.x];        }        __syncthreads();    }    if(row < N && col < N){        C[row * N + col] = val;    }}

Breaking Down the Code

__shared__ float tile_A[TILE_SIZE][TILE_SIZE]; __shared__ float tile_B[TILE_SIZE][TILE_SIZE];

int row = threadIdx.y + blockIdx.y * TILE_SIZE;int col = threadIdx.x + blockIdx.x * TILE_SIZE;

if(row < N && (i * TILE_SIZE + threadIdx.x) < N){         tile_A[threadIdx.y][threadIdx.x] = A[row * N + i * TILE_SIZE + threadIdx.x]; } else {         tile_A[threadIdx.y][threadIdx.x] = 0.0f; }

__syncthreads();

for(int j = 0; j < TILE_SIZE; j++){     val += tile_A[threadIdx.y][j] * tile_B[j][threadIdx.x]; }

if(row < N && col < N)          C[row * N + col] = val;

After all tiles have been processed, the result is written to the output matrix C.

Tile Memory Coalescing and Bank Conflicts

Memory Coalescing: Memory coalescing is crucial for efficient global memory access. In the above kernel, the tiles are loaded into shared memory in a coalesced manner, meaning that consecutive threads access consecutive memory locations, minimizing memory transactions.

Bank Conflicts: Shared memory in CUDA is divided into banks, and when multiple threads access the same bank simultaneously, a bank conflict occurs, leading to serialized access. The kernel is designed to minimize bank conflicts by ensuring that threads access different memory banks during the computation, particularly in the inner loop where matrix elements are multiplied.

Conclusion

By leveraging shared memory and understanding the concepts of memory coalescing and bank conflicts, you can write highly optimized CUDA programs. The MatrixMultiShared kernel demonstrated how tiling can reduce global memory accesses and how careful memory access patterns can avoid bank conflicts. As you continue to explore CUDA, mastering these techniques will be key to achieving maximum performance on your GPU-accelerated applications.

For the full code, you can check out my GitHub repository here. Feel free to experiment with different tile sizes and memory configurations to see how they impact performance!

--

--

Written by Dhanush

Graduate student from CSU- Dominguez Hills. Skilled in CUDA C/C++, parallel programming, and data structures. Proven expertise in React, ASP.NET Core & Docker.

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

Understanding Bottlenecks in Multi-GPU AI Training

Dorha Szasz

Follow

--

Listen

Share

While multi-GPU and multi-node training will always face some overheads compared to single GPU training, ongoing advancements in hardware interconnects, memory architectures, distributed algorithms, and software-level optimizations are steadily chipping away at these bottlenecks to make large-scale AI/ML more efficient. See some examples at the end of the article.

1. The Challenge of Parallelism

Limitations of interconnects like NVLink and PCIe

For example:

2. Gradient Synchronization Overhead

Gradient synchronization is a critical step in multi-GPU training to ensure model consistency. The most common approach involves aggregating gradients from all GPUs after computing them locally. While this ensures the model stays in sync, the time spent in gradient exchange and aggregation often outweighs the benefits of parallel computation.

3. Model Size and Memory Constraints

The size of the AI model plays a pivotal role in determining training efficiency. Large models, such as transformers, require substantial memory for tensors, key-value caches, and gradients. This leads to:

NVIDIA’s DGX servers, which use chips like the GH200 Grace Hopper Superchip, address these issues by eliminating reliance on PCIe technology and offering a faster, more integrated memory bus connecting CPU and GPU memory.

4. Data Communication Across Nodes

When training across multiple nodes, data communication can become the dominant bottleneck. While intra-node communication benefits from fast interconnects like NVLink or NVSwitch, inter-node communication typically relies on Ethernet or Infiniband:

5. Optimizing for Federated Learning

In distributed training scenarios like federated learning, raw data never leaves the local nodes. Only model updates in the form of gradients or weights are shared, greatly reducing the need for frequent data exchange.

This approach mitigates many communication bottlenecks but requires robust optimization algorithms to ensure the global model converges without access to centralized datasets. Techniques like federated averaging help aggregate model updates efficiently.

6. Kernel Fusion for Inference Efficiency

For inference workloads, fusing multiple CUDA kernels together can dramatically improve efficiency. By keeping intermediate computations in local cache/registers rather than writing them back to memory, the need for expensive data movement between the memory and compute units is minimized.

This kernel fusion technique serves latency-sensitive downstream consumers, such as real-time applications, exceptionally well. But it requires careful tuning of the model architecture and operations to identify opportunities for fusion.

Evidence and Solutions

To address these bottlenecks, several technologies and methodologies have been developed:

--

--

Written by Dorha Szasz

Using Deep Learning and Computer Vision to analyze data in multiple domain such as: medical imaging, economics, archeology

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

Ramble in CUDA optimization

Lawliet

Follow

--

Listen

Share

Motivation

I have written my own auto encoder for research purpose since several months ago based on Simoncelli’s 2016 paper. At beginning, I wanted to use some popular Deep Learning frameworks (e.g. Tensor Flow, Caffe2 or MXNet) to do my experiment. However, after several weeks investigation on all these frameworks, I found a very headache problem — extensibility. I’m not saying these frameworks are badly designed, but not allowing users to develop third-part operators just as easily as writing a plug-in is like that you give me a function without any parameters. Then the only way to change the behavior of the function is to modify the source code, this is no doubt a huge engineering stuff because of the poor organization of documents. (It seems to be the common disease of open source software.) Therefore, designing a new framework seems to be the only solution due to the uncommon operator GDN not included in all these frameworks.

GDN

This operator is the core non-linear function in this theory, the expression is as follow (The formula is not important, if you don’t like these god damn symbols, you can simply skip this section.):

The superscript (k) and (k+1) means the layer number, w and u is the input and output which are multi-channels images, the subscript i is the channel number. β and γ are the parameters I want to train. Let’s say we have N channels, then γ is an N by N matrix, β is a N by 1 vector. At the first glance, this function is very similar to Batch Normalization (BN) or Local Reponse Normalization (LRN) which are well supported by cudnn and all Deep Learning framework. But trust me, don’t let your eyes cheat you. It’s very different. (Notice the big division is elementwise division.)

The forward will not consume much computation power, while the backward will eat most power of my GPU. Now let’s look at the backward. There are 3 gradients I need to calculate, ∇β, ∇γ and ∇u.

Oops, I know the feelings when people first look at this because I also want to kill myself when I first look at this monster. But if I can draw a picture for all these shits, you will feel more comfortable.

First, we can easily notice that the input can be regarded as a vector with length m x n. Second, the (blabla…)^(-3/2) appears everywhere in all these gradients. It means that we can just compute this term just 1 time, and cache them for later use. We call this “(blabla…)^(-1/2)” as matrix D . Last, δ is the error propagated to previous layer.

After some simplification, it’s more clear, right? I know a little explanation stilled needed. For the right-handside of the equation, each rectangle is a vector stacked by the matrix as we mentioned above. D is the denominator term in GDN formula, remember the “(blabla…)^(-1/2)” we just mention?

Unlike some advanced algorithms, this computation is so intuitive to most people that we can easily write a CPU program to handle this. And with a little knowledge of CUDA, everyone can port their CPU code to GPU. However, the speed will vary huge differently if you can choose different organizations to launch the kernels.

I call this method “less than just naive” is because this is the first method I ever used. And it almost eats all of my GPU memory even using small size image as input, and achieve slowest performance. Without exploitation any memory reuse, I just copy all these little rectangles both vertically and horizontally to get bigger matrices as shown in the below picture, and launch a lot of kernels organized in 1-D. Then sum them together.

The only advantage of this algorithm is that you don’t need to calculate the index in each CUDA thread, because the thread id is just corresponding the memory index uniquely. So all you need to do is some multiplication, and then use cublas to sum each little colored rectangle by dot product with 1-vector(a vector that is fulled with all ones.). But as you see, the size of rectangle is not as small as I just draw here, the size is equal as an image. And for each vector in this picture, the size will be N x N x imageSize x batchSize. And obviously, we waste (N-1) x N x imageSize x batchSize x 4 bytes, not to say the time wasted on accessing all these redundancy global memory.

2. Naive Algo.

For the 1st algorithm, I can only train less than 4 images with size 128 x 128 in my network each iteration, and the time is almost 2 seconds. (My GPU is GTX 1080.) This reality forced me to improve my algorithm, otherwise, I have to wait for almost 2 months to get my results.

Because the number of kernels I need to launch is definitely much more than the CUDA cores in my GPU, so no matter what method I use, the cuda driver will serialize these tasks. Then I decide not to copy all these memories. Instead, I will launch the N x imageSize kernels organized in 1-D for N times (N is the total number of channels ).

One can see that the improvement is obvious. Because, we don’t need to replica the data tons of times any more. The global memory access in GPU is very expensive. The memory access pattern is also easy, because when you get the thread id, by using just one mod operation you can get the memory index ( memory index = thread id % imageSize). However, in this method, because the kernels are still organized in 1-D, and we use a for-loop to launch all these kernels, then we may not be benefit from the smarter schedule algorithm by GPU, although I have already tasted blood. Now 2 months of training time can be shrunken to almost 2 weeks by this little change.

3. Smarter organization Algo.

So far now, I still haven’t considered about the power of share memory, because for me, usually design a good kernel pattern is boring and headache. Obviously, 1-D kernel pattern is the easiest code to write. However, better performance deserves more careful design. The algorithm in this section, to my surprise, achieves 3x speed up to the second algorithm.

Back to the Fig 1., one can see that the first row of first 3 right-handside matrices δ0, w0 and D0, are the same. Therefore, we can calculate one row of γ in one block, for each block we can launch imageSize of threads, and for each thread we can compute all channels by using a for-loop.

So from Fig 5., it’s very intuitive to put δ0, w0 and D0 in share memories, and for thread i, it goes from 0 to N-1 to read one pix of N channels to multiply with δ0, w0 and D0 in the share memories. The pseudocode is as follow:

blockId = blockIdx.x; threadId = threadIdx.x;shareDelta <- delta[blockId];  shareW <- W[blockId];shareD <- D[blockId];_synchronize();for(i = 0; i < N-1; i++){   result[threadIdx i*imgSize] = shareDelta[threadId] *                                 shareW[threadId] *                                 shareD[threadId] *                                  W[threadId + i*imgSize];}

The selection of row major computation instead of column major computation as Algo 2 is because that to computation one row in one grid, we can share 3 vectors δ0, w0 and D0. But if we compute one column as in Algo, we can only share 1 vector w0. (Again, see Fig 1.).

In this code snippets, there is no if … else … block. This is very important in parallel computing. Because all threads are parallelly running, the ideal situation is that all these threads finish their jobs at the same time. But if there is if … else … block, the branches will make these threads do the different tasks so that they will finish at different time. Then the compute time will be determined by the slowest thread(s).

No index computing is also an advantage. By design the 1-D pattern, we have to use thread id to compute the memory index, but here, there is no need to convert the blockId and threadId to 1-D memory index to access the data.

Last, because my data is stored in the column major, it means that, like vector δ0, all the elements in this vector are contiguously stored. So it’s benefit from the global memory coalescing mechanism. Global memory is also an important concept in cuda.

In the aspect of hardware, there are 16 cuda kernels are organized in one warp. When one of the thread accesses data, say a1 in above picture, the data bus will not only transfer a1, but also transfer a1~a32 to the cache in order to accelerate the data accessing for the other 15 kernels. Therefore, when I read global data to share memory, each 32 bytes I just read once, and all the others are read from cache which is several hundred faster. Thanks to the space and time locality theory.

4. A little more improvement

Today, I suddenly notice that actually I don’t need any share memory, but can use const memory. Because for vectors δ0, w0 and D0, each thread in one block only need to access once. So before the for-loop, we actually can cache the element in the const memory. And another sugar is because that each thread only access one element, no thread synchronization is needed.

The code is as follow:

blockId = blockIdx.x; threadId = threadIdx.x;const float constDelta = delta[blockId * imgSize + threadId];  const float constW = W[blockId * imgSize + threadId];const float constD = D[blockId * imgSize + threadId];for(i = 0; i < N-1; i++){   result[threadIdx + i*imgSize] = constDelta * constW *                                   constD *                                    W[threadId + i*imgSize];}

From the code above, we can see that constDelta, constW, constD can be reused for N times from local memory which is always stored in local register. Therefore, the bandwidth is more than share memory.

Reduce Operation

All the algorithms I have talked about are not completed, because all I got from above algorithms are actually raw γ as shown below:

I need to accumulate each vector on the left-handside to get one element. The first choice is the cublas API, cublasSsbmv. This function will do the matrix-vector multiplication. So we can regard the left-handside vectors as a matrix, and multiply it with an all ones vector to get one row of gradient of γ. And repeat it N times to get final result. But I notice that there are other API cublasSgemmBatched. This function can do the batch matrix-vector multiplications. Then I did an experiment to test which one is faster:

for-loop of N matrix-vector multiplications VS batch matrix-vector multiplications.

The result shows that the for-loop is much faster. I don’t know the reason however, maybe it’s because my N here is too small (N = 256).

I will not show how to compute ∇β and ∇u, because they’re similar to ∇γ. I know there must be further optimization or better design than I have. CUDA optimization is usually difficult for people don’t deeply understand the organization of GPU. Programmers who are familiar with CPU are always benefit from modern OS and powerful compiler. However, GPU is much different and sophisticated than CPU on writing sufficient code although it’s much convenient than using graphic shaders to do computation previously. It still needs several years to perfect the ecological environment.

--

--

Written by Lawliet

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

Features of NVIDIA CUDA

Abhishek Nandy

Follow

--

Listen

Share

As our startup Dynopii got selected for the Nvidia inception program I started exploring the libraries here is a small walkthrough on features of NVIDIA CUDA.

CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model developed by NVIDIA for general purpose computing on GPUs (graphics processing units). CUDA enables developers to leverage the massive parallelism of GPUs for a wide range of applications, including machine learning, computer vision, and scientific computing.

In recent years, NVIDIA has introduced several new features and improvements to the CUDA platform, including:

CUDA Graphs: CUDA Graphs is a new programming model for CUDA that allows developers to express the dependencies between CUDA kernels in a more explicit and flexible manner. CUDA Graphs makes it easier to write complex, data-parallel algorithms that can take advantage of the full parallelism of the GPU.

CUDA Dynamic Parallelism: CUDA Dynamic Parallelism is a feature that allows CUDA kernels to launch other CUDA kernels dynamically, without the need for explicit host-device memory transfers. This feature enables a new class of algorithms that can adapt to the characteristics of the data being processed, leading to more efficient use of GPU resources.

CUDA Streams: CUDA Streams is a feature that allows multiple CUDA kernels to execute concurrently on the GPU. This can be used to overlap data transfers and kernel execution, resulting in better GPU utilization and improved performance.

CUDA Libraries: NVIDIA provides a set of libraries that are optimized for CUDA and provide common functionality such as linear algebra, fast Fourier transforms, and image processing. These libraries are designed to be easy to use and can significantly reduce the time and effort required to write CUDA code.

CUDA Toolkit: NVIDIA provides a comprehensive development environment for CUDA called the CUDA Toolkit. This includes a compiler, libraries, debugging and profiling tools, and documentation. The toolkit is designed to make it easy for developers to write, debug, and optimize CUDA code.

CUDA on ARM: NVIDIA has announced that it is working on a version of CUDA that will run on ARM-based processors. This will enable developers to run CUDA code on a wide range of devices, including mobile phones, tablets, and embedded systems.

CUDA on Multi-GPU systems: CUDA allows to use of multiple GPU for a single application. This can be used to scale the performance of an application beyond what is possible with a single GPU, and it can also be used to add redundancy to a system for improved reliability.

CUDA on Cloud: NVIDIA provides cloud-based GPU instances that can be used to run CUDA code. This allows developers to easily scale up their applications to run on large clusters of GPUs without the need to purchase and maintain the hardware themselves.

CUDA Graphs

An example of a CUDA graph could be a series of image processing operations that need to be performed on a large set of images. The operations could include tasks such as image resizing, filtering, and feature extraction. The CUDA graph for this example could be organized as follows:

A CUDA kernel that resizes the images to a smaller size is the first node in the graph.The second node in the graph is a CUDA kernel that applies a filtering operation to the resized images.The third node in the graph is a CUDA kernel that extracts features from the filtered images.The final node in the graph is a CUDA kernel that performs image classification on the extracted features.Each of these nodes in the graph represents a CUDA kernel and the edges between the nodes represent the dependencies between the kernels. For example, the image resizing kernel must complete before the filtering kernel can begin, and the filtering kernel must complete before the feature extraction kernel can begin.

When the CUDA graph is launched, the CUDA runtime will automatically manage the execution of the kernels and the transfer of data between them, ensuring that dependencies are satisfied and that the kernels are executed in the most efficient order. The CUDA graph API allows developers to launch the graph and manage resources such as memory and stream.

This is just an example, the CUDA graph can be more complex and can include multiple branches and loops, based on the task and the specific use case.

complete code walkthrough CUDA graphs

A complete code walkthrough of a CUDA graph would be quite extensive, as it would involve the creation and execution of multiple CUDA kernels, as well as the management of memory and other resources. However, I can provide an overview of the main steps involved in creating and executing a CUDA graph using the CUDA Graphs API:

Include the necessary CUDA libraries and headers, such as cuda_runtime.h and cuda_graph.h.

Create a CUDA graph handle using the cudaGraphCreate() function. This handle will be used to manage the graph throughout its lifetime.

Create CUDA graph nodes for each of the kernels that will be included in the graph. This can be done using the cudaGraphAddKernelNode() function.

Create edges between the nodes to represent the dependencies between the kernels. This can be done using the cudaGraphAddDependencies() function.

Allocate memory and other resources required by the kernels, such as CUDA streams and event handles.

Launch the CUDA graph using the cudaGraphLaunch() function. This will execute the kernels in the graph and manage the transfer of data between them.

Wait for the graph to complete execution using the cudaGraphSynchronize() function.

Release the resources allocated for the graph and free the CUDA graph handle using the cudaGraphDestroy() function.

Here is an example of a simple CUDA graph that performs a vector addition on two arrays:

#include <cuda_runtime.h>#include <cuda_graph.h>// CUDA kernel for vector addition__global__ void vecAdd(float* a, float* b, float* c, int N) {    int i = threadIdx.x + blockIdx.x * blockDim.x;    if (i < N) c[i] = a[i] + b[i];}int main() {    // Allocate memory for input and output arrays    float* a, *b, *c;    cudaMalloc(&a, N * sizeof(float));    cudaMalloc(&b, N * sizeof(float));    cudaMalloc(&c, N * sizeof(float));    // Fill input arrays with data    // ...    // Create CUDA graph handle    cudaGraph_t graph;    cudaGraphCreate(&graph);    // Create CUDA graph nodes for kernel and memcpy    cudaGraphNode_t kernelNode, memcpyNode;    cudaGraphAddKernelNode(&kernelNode, graph, vecAdd, 0, 0, 0);    cudaGraphAddMemcpyNode(&memcpyNode, graph, cudaMemcpyDeviceToHost, c, 0, 0, 0);    // Create edges between nodes to represent dependencies    cudaGraphAddDependencies(graph, kernelNode, &memcpyNode, 1);    // Launch the CUDA graph    cudaGraphLaunch(graph, 0);    // Wait for the graph to complete execution    cudaGraphSynchronize(graph);    // Release resources    cudaGraphDestroy(graph);    cudaFree(a);    cudaFree(b);    cudaFree(c);    return 0;}

That's a small start with CUDA.Let’s meet next time.

--

--

Written by Abhishek Nandy

AI Consultant |Intel Certified oneAPI Instructor|Thinker|Innovator |Bioinformatics AI|Rapid Development Specialist

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

Apra Labs Blog

Home

About

Technology company focussed on computer vision, machine learning and cloud.

Basics of writing a fast CUDA Kernel

Sai Chaitanya E

Follow

Apra Labs Blog

--

Listen

Share

For one of our clients, we have developed an AI-powered Visual Inspection system. A single device handles multiple camera feeds, each running at over 250 fps. Each frame goes through a series of detection and classification modules. The camera feed is also streamed on demand via RTSP after applying rescale, image effects (brightness, contrast, hue, saturation adjustment) and logo overlay.

In short, each module has to be very very fast. Below we show you how we have made the image effects module faster to meet performance goals. Lets first see these operations in action.

Image Effects

In RGB Color space, brightness and contrast adjustment can be modeled as g(i,j)=α⋅f(i,j)+β

And in HSV Color space, hue and saturation adjustment can be modeled as

The parameters α>0 and β are constants.

NVIDIA Performance Primitives (NPP)

The input image was in YUV420 color space. To apply the image effects using the NPP library, we have to do the below 10 operations in sequence.

YUV420 color space (Input) → Convert to RGB color space → Contrast Adjustment → Brightness Adjustment → Convert to HSV color space → Split Channels → Hue Adjustment → Saturation Adjustment → Merge Channels → Convert to RGB color space → Convert to YUV420 color space (Output)

For contrast and saturation the input values can be float, but the NPP MulC function accepts only uint8_t values.

The 10 NPP functions are called in sequence and synchronized at the end one time, which internally launches 10 CUDA kernels, to put it in simple terms each pixel is read and updated (write) 10 times which is not ideal.

The total time taken in microseconds averaged over 10000 calls and profiled using nvprof is 2057 microseconds.

So, the reason for writing a custom CUDA kernel instead of using NPP library

1. float multiplication

2. Do all the operations in a single pass.

Matching Performance

We wanted to spend a day to see if we can write a CUDA kernel to get better performance. To learn the tricks of profiling and writing fast CUDA kernels, we started with a simple multiplication kernel and match NPP performance.

g(i,j)=α⋅f(i,j)

We first started with a basic loop — each thread operates on 1 pixel. The image is 8 bit and width is 32 byte aligned, the CUDA kernel block size is 32x32. The time taken was ~3x of MulC.

So we started following the performance guide. Performance optimization revolves around three basic strategies:

The most useful for our scenario was (2) from the above.

We changed each thread to operate on 4 pixels — this way each warp (32 threads) will access 128 bytes. Now, the time taken is comparable to MulC. Additionally we are using CUDA math library wherever applicable.

We compared the time taken in microseconds averaged over 10000 calls and profiled using nvprof.

Conclusion

With the insight gained, we wrote a CUDA kernel which does all the operations in the same thread. The time taken reduced from 2057 microseconds to 447 microseconds, roughly 5x faster compared to calling NPP functions in sequence. Also, the number of intermediate CUDA memory buffers required is 0 as compared to 5 with NPP because we are doing the computations in a single pass.

So, to write a fast CUDA kernel, one of the important step is to review and optimize memory usage to achieve maximum memory throughput.

--

--

Published in Apra Labs Blog

Technology company focussed on computer vision, machine learning and cloud.

Written by Sai Chaitanya E

Architect — Machine Learning Solutions, Apra Labs

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

Building fused CUDA kernels for RNNs

kevin zhang

Follow

--

1

Listen

Share

Introduction

One day, you might decide to implement your own RNN. PyTorch offers a convenient way to do this using the torch.nn.functional module. For example, here is how we’d implement the forward pass of GRU:

I won’t walk through the code for the forward pass, since this is not the purpose of the post. It suffices to say that the underlying mathematics for this implementation is the same as that of the PyTorch GRU module. So we would expect the performance of the two to be similar, right?

Not quite. Here are the average times it take for the forward passes from each implementation to execute on GPU:

PyTorch Library: 93 μs. Python: 332 μs.

Several factors contribute to the performance gap. First, the operations in our forward pass implementation don’t know about each other, which means that PyTorch must execute the operations individually. Since each individual call to the kernel of an operation, which may involve launch of a CUDA kernel, has a certain amount of overhead, this overhead may become significant across many function calls. Second, the Python interpreter itself can slow down our program.

On the other hand, the PyTorch library implements its GRU forward pass as a fused kernel. This means that multiple operations in the forward pass are placed into the same kernel, which results in fewer kernel calls. The library also implements the kernel with CUDA to take advantage of the parallelism GPUs provide.

Therefore, a reliable way to optimize the speed of a custom RNN is to write its own fused CUDA kernel. PyTorch has an official tutorial on how to do this, so I won’t repeat the details. However, the tutorial assumed readers to have an understanding of GPU programming and didn’t explain the underlying logic of the RNN CUDA kernels. II’d like to fill in this knowledge gap.

CUDA Optimization

We will continue using the forward pass of GRU as our implementation example.

First, note that we can rewrite lines 12–15 from the python implementation as:

Two important properties of the right hand side are 1) All the constituents (i_n, i_r, i_i, h_n, h_r, h_i ) are vectors of the same dimension; and 2) all the operations are pointwise. It follows that the operations for a given index is independent from the operations of other indices. This principle can be visualized in the following diagram, where the red arrow suggests that the operations on the second index happen independently. We will call a such an index a pointwise index.

The same principle applies when batch number is greater than 1: The pointwise operations of a particular index for a particular batch is independent from all other operations:

We can take advantage of this fact by parallelizing the vector operations across multiple threads on a GPU — one thread for each index computation. This is the key to why GPU programming is so effective for RNNs, since pointwise operations are prevalent in most recurrent architectures.

Let’s see how this is done. In CUDA programming, the GPU is conceptually broken down into blocks of threads. Here’s a visualization of 4095 blocks, with 255 threads in each block.

The dimension of blocks can be more than 1. Here’s what having a block dimension of 2 (a grid of blocks) look like:

Now we need to assign the computation for each pointwise index to a particular thread. Let’s dive into the CUDA code for the GRU forward pass to see how this is done.

The code above consists of two functions. gru_cuda_forward is the entry point, it is executed on CPU. It calls the kernel function gru_cuda_forward_kernel, which is executed on GPU. Kernel methods have the __global__ key words in their declarations. I’ve left out the implementation for the kernel function, and will come back to it later.

Since gru_cuda_forward is the entry point, we will go over it first.

Lines 12–19 are essentially the same as lines 4–7 from our PyTorch implementation. The implementation for the matrix multiplication operator torch::addmm is highly optimized, so we make use of it. Lines 18–19 simply reshape our results into a format that will be easier to work with.

Lines 21–24 initialize vectors, including the new state, that the kernel function will populate later.

Line 26–27 partition the GPU into a grid of (m x n) blocks, each block containing 1024 threads. The horizontal grid dimension n equals the batch number, while the vertical grid dimension m is the minimum number of blocks, when concatenated together, required to match or surpass the state size, given that the single unit of length is the thread. This way, each pointwise index can be mapped to an unique thread:

A subtle point is that due to the way the number of blocks is calculated on line 27, the total threads in a column may be greater than the actual state size. This is acceptable since it has no harmful effects. However, the reverse does not hold— if columns contains less threads than the state size, then there will be less threads than pointwise indices, making the reverse mapping from threads to every pointwise index impossible.

On lines 29–38, the kernel function is called, and is executed on all the threads on the grid simultaneously. Inside the kernel function, the programmer has access to the Block and Thread ID of the current thread, which can be used to reverse map the thread to its pointwise index.

All the inputs to the kernel function are represented as single dimensional vectors, this makes indexing a bit convoluted. For example, in order to access the element gate[batch][column] inside the kernel, we need to do gate[batch index * state_size + column].

Besides being verbose, this expression needs additional variables such as state_size being passed into the kernel as arguments.

Fortunately, ATen provides a way to index into a tensor efficiently without having to convert to a single pointer through packed accessors. On lines 31–37, the input vectors are transformed into accessors. Let’s dissect the last input accessor on line 37:

new_h.packed_accessor<scalar_t,2,torch::RestrictPtrTraits,size_t>())

Here, an accessor is created for new_h, while asserting that it is a 2 dimensional tensor with scalar elements. This allows us to callnew_h[batch][column] inside the kernel.

Now, let’s take a look at the kernel function itself:

It takes in accessors as input arguments. On lines 11 and 13, the block and thread IDs are mapped to the pointwise index (which consists of the batch index and column index). Hopefully, the calculation for the column index makes sense given that blockDim.x equals to the number of threads in a block.

Line 15 ensures that computation only takes place if the column index is smaller than the state size. This check is necessary since the number of threads in a column may be greater than the state size. Inside the conditional, we simply compute the operations from equation 1 for the pointwise index, and places the answers into the output vectors at that index.

Once all the threads complete their kernel executions, the output vectors are returned back to the caller.

That’s it! We have essentially optimized our forward pass by 1) fusing multiple operations into the same kernel; and 2) parallelizing the kernel function across multiple threads on a GPU.

Performance Comparison

Our hope was to speed up our forward pass by building for it a customized fused CUDA kernel. Let’s see if that holds true.

Here’s the average time it takes for the forward pass to execute on GPU for various implementations:

PyTorch Library: 93 μsCUDA: 178 μsPython: 332 μs

The CUDA optimization yielded approximately 100% speed increase over the python implementation. But the interesting thing to note is that the customized kernel still lags behind the PyTorch library in performance.

I plan to investigate the cause for the difference in the coming weeks, and will follow up with another post if the investigation yields fruitful results. Meanwhile, I hope this article helps you understand how GPU programming can help speed up RNNs, both in principle and in practice.

The code used in this tutorial can be found here.

--

--

1

Written by kevin zhang

Responses (1)

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

Boosting Performance with CUDA: A Deep Dive into Vector Addition on GPU vs. CPU

Dhanush

Follow

--

Listen

Share

Introduction

In the world of high-performance computing, the ability to process large datasets efficiently is paramount. Graphics Processing Units (GPUs) have become a go-to solution for parallel computing tasks due to their massive parallelism, especially when compared to traditional CPUs. This article explores the performance benefits of using GPUs for vector addition, a fundamental operation, by comparing it to the CPU.

Understanding the Code

The code snippet provided demonstrates how to perform vector addition on both the CPU and the GPU. Here’s a step-by-step breakdown:

1. Vector Initialization

The code begins by initializing two vectors, one and two, each containing n integers. These vectors will be added together to produce a third vector, sum_of_two, which stores the results.

2. CPU Vector Addition

Before diving into the GPU implementation, the code performs vector addition on the CPU to provide a baseline for comparison. This is done using a simple loop:

for (int i = 0; i < n; i++) {    sum_of_two_cpu[i] = one[i] + two[i];}

This loop adds corresponding elements of one and two and stores the result in sum_of_two_cpu.

3. Memory Allocation on GPU

To utilize the GPU, memory must be allocated on the device. The following lines of code allocate memory for the vectors on the GPU:

cudaMalloc((void**) &dev_a, n*sizeof(int));cudaMalloc((void**) &dev_b, n*sizeof(int));cudaMalloc((void**) &dev_c, n*sizeof(int));

Here, cudaMalloc allocates memory on the GPU for vectors a, b, and c, which correspond to one, two, and sum_of_two on the CPU.

4. Data Transfer from Host to Device

Next, the data from the host (CPU) needs to be copied to the device (GPU). This is done using cudaMemcpy:

cudaMemcpy(dev_a, one.data(), n*sizeof(int), cudaMemcpyHostToDevice);cudaMemcpy(dev_b, two.data(), n*sizeof(int), cudaMemcpyHostToDevice);

This step is crucial because, unlike the CPU, the GPU operates on its own memory space.

5. GPU Vector Addition

The vector addition on the GPU is performed by the add_vec kernel function:

__global__ void add_vec(int* a, int* b, int* c, int n) {    int tid = threadIdx.x + blockIdx.x * blockDim.x;    if(tid < n){        c[tid] = a[tid] + b[tid];    }}

Each thread in the GPU is responsible for adding a pair of elements from the vectors a and b, and storing the result in c. The kernel is launched with a specific number of threads per block and blocks per grid, calculated as:

int threads_per_block = 256;int blocks_per_grid = (n + threads_per_block - 1) / threads_per_block;

6. Timing the Operations

To compare the performance, the code measures the time taken by both the CPU and GPU to complete the vector addition:

auto start_cpu = high_resolution_clock::now(); auto end_cpu = high_resolution_clock::now();

cudaEvent_t start_gpu, stop_gpu;cudaEventCreate(&start_gpu); cudaEventCreate(&stop_gpu); cudaEventRecord(start_gpu);

7. Data Transfer from Device to Host

After the computation, the result from the GPU is transferred back to the host:

cudaMemcpy(sum_of_two.data(), dev_c, n*sizeof(int), cudaMemcpyDeviceToHost);

8. Memory Deallocation

Finally, the code deallocates the memory on the GPU:

cudaFree(dev_a);cudaFree(dev_b);cudaFree(dev_c);

Why GPUs Excel in Large-Scale Operations

GPUs are designed to handle thousands of threads simultaneously, making them ideal for tasks that can be parallelized, such as vector addition. The key reasons for their superior performance in large operations include:

Conclusion

This code demonstrates the power of GPUs in handling large-scale operations like vector addition. By leveraging CUDA, you can achieve significant performance gains, especially as the size of the dataset increases. Understanding how to manage memory transfer between the CPU and GPU and efficiently utilizing GPU resources is key to unlocking these performance benefits.

For those interested in exploring the code further, you can find the complete implementation on GitHub: CUDA Vector Addition — CPU vs. GPU.

Feel free to tweak the content to match your writing style.

--

--

Written by Dhanush

Graduate student from CSU- Dominguez Hills. Skilled in CUDA C/C++, parallel programming, and data structures. Proven expertise in React, ASP.NET Core & Docker.

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

How to Increase Computational Efficiency for PReLU in CUDA — OneFlow Performance Optimization

OneFlow

Follow

CodeX

--

Listen

Share

Written by Zekang Zheng; Translated by Yanjun Hu, Jiali Shen

PReLU is an activation function that is frequently used in InsightFace. It has two operating modes: PReLU(1) and PReLU(channels). For the latter, PReLU is equivalent to a binary broadcast operation. In this article, we are going to talk about optimizing the broadcast operations in CUDA.

PReLU is an activation function that is frequently used in InsightFace. It has two operating modes:

InsightFace adopts the second mode of PReLU. We wrote about how to optimize CUDA elementwise operationsbefore. Today, we are going to talk about optimizing the broadcast operations in CUDA.

1 Naïve implementation

Here is a naïve implementation. Firstly, get the index of the current element-“x” in the loop. Secondly, infer the index of the corresponding alpha weight. Thirdly, return results by seeing if “x”>0: if “x”>0, return “x”; if “x”<0, return “alpha*x”. The code is as follows:

Note:

In CUDA, integer division comes with high computational costs. As you can find in the CUDA Toolkit Documentation (Chapter 5.4.1 Arithmetic Instructions):

Integer division and modulo operation are costly as they compile to up to 20 instructions.

Calculating the index of alpha involves one integer division and one modulo operation. These computations represent half of the workload of the kernel. Thus, in what follows, we are going to tell you how to reduce integer divisions and modulo operations while increasing read/write bandwidth using a vectorized method.

2 Optimizing by PackType vectorization

Here is a simple case: if the tensor shape being input is (1,2,4,4), then its operating mode will be PReLU(2).

What’s obvious is that the input on H and W dimensions is continuous. In this situation, if inner_size is divisible by pack size, the elements in a pack will be applied to the same alpha weight, just as the following figure shows:

In this way, vectorization operations are advisable to improve the read/write bandwidth utilization. And the elements in each pack only need to be calculated once, which cuts down considerable computations, since we no longer need to calculate the elements one by one. The code is as follows:

Let’s compare the results before and after an optimization operation offered by Nsight Compute. After testing data (96, 64, 112, 112) on the A100–40GB GPU, we got the performance results of two kernels as shown in the following figure: the blue one is optimized by vectorization, while the green one is naively implemented. It’s clear that an optimization operation efficiently reduces 20%-30% of calculations and boosts throughput by 30%. Besides, the optimized kernel’s bandwidth is up to 1350GB/s, which gets very close to A100’s theoretical limit.

However, not all tensor shapes support a vectorization operation. If the inner_size cannot be divisible by its corresponding pack_size, the shape can only settle for a naive implementation.

3 Benchmark

When conducting a benchmark test on the NVIDIA A100–40GB GPU, we compared the performance of OneFlow and PyTorch when they deal with different tensor shapes in the InsightFace library. And the testing result is as follows:

We can see that OneFlow, empowered by the optimized activation function-PReLU, delivers better performance than PyTorch by nearly 200% in most conditions. As for the last testing example, the tensor shape is so special that a vectorization optimization doesn’t work, so OneFlow performs on a par with PyTorch.

Related articles：

Welcome to visit OneFlow on GitHub and follow us on Twitter and LinkedIn.

Also, welcome to join our Discord group to discuss and ask OneFlow related questions, and connect with OneFlow contributors and users all around the world.

--

--

Published in CodeX

Everything connected with Tech & Code. Follow to join our 1M+ monthly readers

Written by OneFlow

OneFlow is a deep learning framework designed to be user-friendly, scalable and efficient. https://github.com/Oneflow-Inc/oneflow

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

CUDA: Shared memory

Rustam

Follow

--

Listen

Share

CUDA shared memory is a type of memory accessible to all threads within the same block. It resides on the GPU chip itself, making it significantly faster to access compared to off-chip global memory.

Memory size depends on a GPU architecture and configuration. For example, in a GPU (compute capability 8.6), based on the Ampere architecture, the shared memory capacity per SM is up to 100 KB and maximum shared memory per thread block is 99 KB.

The maximum amount of shared memory available per thread block is typically smaller than the maximum shared memory partition available per Streaming Multiprocessor (SM). 1 KB of Shared Memory is reserved for system use and is not made available to thread blocks.

Dynamic & Static

__shared__ int sharedMem[256];

But static memory allocation has a restriction: one cannot allocate over 48KB of Shared Memory, it will lead to compile time error: “uses too much shared data”.

If you need multiple dynamically allocated memory regions within a block, you must use pointers to offsets within a single shared memory allocation.

// kernelextern __shared__ int sharedMem[];int* v1 = &sharedMem[0];int* v2 = &sharedMem[10];...// specify a memory size as the 3rd parameter on kernel launchkernel<<<BLOCKS, THREADS, 20 * sizeof(int)>>>(...);

Also, if you need over 48KB dynamic Shared Memory, you need to call a method cudaFuncSetAttribute before launching a kernel:

cudaFuncSetAttribute(KERNEL,   cudaFuncAttributeMaxDynamicSharedMemorySize,   SIZE);

Occupancy

When configuring an application, it’s crucial to consider the resources available on each Streaming Multiprocessor (SM). One important resource to manage is a Shared memory, which is distributed among thread blocks residing in the SM. If a thread block requires more shared memory than is available on an SM, that block won’t be scheduled and will wait for the first free SM where it can reside.

Warp & Banks

Shared memory in CUDA consists of 32 banks, organized such that successive 32-bit words map to successive banks. Each bank has a bandwidth of 32 bits per clock cycle.

A warp is a group of 32 threads within the same thread block. When a thread attempts to access shared memory, the access pattern across the threads in the warp can impact memory performance due to bank conflicts.

Bank Conflict

A bank conflict can occur only in the same warp

A bank conflict occurs when multiple threads within the same warp attempt to access memory locations that belong to the same memory bank simultaneously. This can lead to serialization of memory accesses, reducing memory bandwidth utilization and potentially slowing down the kernel’s performance.

Example (8 banks, warp size = 8, data size = 2x8):

To fix it when accessing a 2D array by column, we can add an extra column to the matrix. While this column doesn’t contain useful data for our calculations, it helps to ensure that each thread within a warp accesses data from different memory banks.

Nvidia Nsight

A simple example, that attempts to access to columns with a stride = 8:

sharedData[block.thread_rank() * 8] += data[globalId];

You can detect and analyze bank conflicts using Nvidia Nsight Compute. In the image below, you can observe that the source code exhibits 8-way bank conflicts, meaning each thread in a warp accesses to some banks 8 times. This results in increased load/store wavefronts, causing serialization and processing on different cycles, each occurring 8 times.

Wavefront is the maximum unit that can pass through that pipeline stage per cycle. If not all cache lines or sectors can be accessed in a single wavefront, multiple wavefronts are created and sent for processing one by one, i.e. in a serialized manner

A fixed code block:

sharedData[block.thread_rank()] += data[globalId];

Broadcasting

When 2 or more threads in a warp access to the same address in a bank, it will not result in a bank conflict. The data will be broadcasted with no affect to the performance.

More than the word size

When dealing with data elements larger than the word size (4 bytes), such as double precision floating-point numbers (8 bytes), the hardware still needs to ensure proper access to the data, and this may involve multiple transactions. Each transaction fetches a chunk of data, often referred to as a “cache line” or “coalesced access unit.”

You don’t need to care about that, hardware will do it itself.

The same applies if the data size is less than the word size (< 32 bits).

L1/Shared Memory

Shared Memory shares on-chip storage with the L1 cache. But Shared memory is explicitly controlled by the programmer and used for inter-thread communication and data sharing, while the L1 cache is managed by the GPU hardware and helps improve memory access latency and bandwidth by caching data and instructions fetched from global memory.

CUDA provides an API to set the carveout, i.e., the preferred shared memory capacity:

cudaFuncSetAttribute(kernel_name,   cudaFuncAttributePreferredSharedMemoryCarveout,   carveout);

It is considered a hint by the driver, it may choose a different configuration, if needed. The driver may choose a different configuration if required to execute the function or to avoid thrashing. Don’t use it if you are not sure, increasing carveout may lead to decreasing L1 hit rate, consequently — result in a worse performance.

Conclusion

Thanks for reading, feel free to comment with corrections or ideas. And don’t forget to clap if you like this article :)

References

https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html

NVIDIA Ampere GPU Architecture Tuning Guide

The programming guide for tuning CUDA Applications for GPUs based on the NVIDIA Ampere GPU Architecture.

docs.nvidia.com

https://docs.nvidia.com/nsight-compute

--

--

Written by Rustam

My interests are C++, ML and CUDA. github: https://github.com/fatlipp

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

Run Your Python User Defined Functions in Native CUDA Kernels with RAPIDS cuDF

Combining Python/CUDA JIT Compilation for Flexible Acceleration in RAPIDS

Jiqun Tu

Follow

RAPIDS AI

--

1

Listen

Share

In this blog, we’ll introduce our design and implementation of a framework within RAPIDS cuDF that enables compiling Python user-defined functions (UDF) and inlining them into native CUDA kernels. Our framework uses the Numba Python compiler and Jitify CUDA just-in-time (JIT) compilation library to provide cuDF users the flexibility of Python with the performance of CUDA as a compiled language. An essential part of the framework is a parser that parses a function in one of the CUDA intermediate representations stage, which is compiled from the Python UDF, into an equivalent CUDA device function that can be inlined into native CUDA C++ kernels. Our approach makes it possible for Python users without CUDA programming knowledge to extend optimized dataframe operations with their own Python UDFs, which enables more flexibility and generality for high-performance computations on dataframes in RAPIDS.

We start by giving examples on how to use the feature, followed by the goals we intend to achieve. Finally, we explain how things work in the background to make the function possible.

How to Use the Feature

The feature is built into the framework of RAPIDS cuDF and is easy to use. Once a dataframe is created, simply call the interfaces that support this feature with the user-defined Python function. Currently, the list of support includes:

In the following, we give examples with applymap and rolling.

The `applymap` example:

>>> import cudf>>> import cudf.core>>> from cudf.core import Series>>> import numpy as np>>> a = Series([9, 16, 25, 36, 49], dtype=np.float64)>>> a.applymap(lambda x: x ** 2)0      81.01     256.02     625.03    1296.04    2401.0dtype: float64>>> a.applymap(lambda x: 1 if x in [9, 44] else 2)0    11    22    23    24    2dtype: int64

The `rolling` example:

>>> def foo(A):...     sum = 0...     for a in A:...         sum = sum + a...     return sum... >>> a.rolling(3, 1, False).apply(foo)0      9.01     25.02     50.03     77.04    110.0dtype: float64

What the Feature Intends to Achieve: Flexibility and Performance

Ahead-of-Time Compilation

Traditionally, with ahead-of-time compilation, CUDA kernels are compiled into SASS machine-level code at compile-time and launched at run time. In cases where operator functions need to be called by kernels, the use of function pointers or stack frame, which usually jeopardizes performance, are avoided by inlining the operator function, as shown in the following code:

Performance is achieved; however, at the price of flexibility. Often at compile-time, the operator function is not known. In most cases, the program does not reach the end-users until run time, and it is the users who are going to decide what operator function is needed. With ahead of time compilation, users cannot write their operator function without recompiling the whole program while still having the maximum performance.

Just-in-Time Compilation

Just-in-time (JIT) compilation, or run-time-compilation, helps. Utilizing CUDA runtime compilation (NVRTC) and the JITIFY library, the code string of the operator function, written at run time, can be inlined into the code string of the kernel (before the combination is compiled at run time) and launched with the same performance of a corresponding traditional native CUDA kernel. Flexibility and performance are optimized, with the only overhead being the time needed to perform the run time compilation.

Combine Python and CUDA

Combining the flexibility of Python as an interpreted language and the performance of CUDA as a compiled language will have broader coverage. A Python UDF can be written, without the knowledge or even awareness of CUDA, compiled and inlined into carefully optimized pre-defined CUDA kernels and launched on GPUs with maximum performance, as shown in the usage examples.

For more information about how Python is added to the workflow on top of NVRTC/JITIFY framework, check out my NVIDIA DevBlog on the topic.

A Performance Benchmark of `applymap`

We compare the performance of pandas.apply with cudf.applymap for dataframes with large numbers of rows and the later one is able to achieve significant speedup over the former one. The following benchmark is measured on an Intel(R) Xeon(R) Gold 6128 CPU and an NVIDIA Quadro GV100 GPU. Note that these results do not include the overhead of JIT compilation. This overhead is a one-time cost paid only on the first execution of this feature using a specific UDF.

Conclusion

Utilizing the benefits of Python and CUDA, the combined Python/CUDA JIT compilation in RAPIDS cuDF allows users to apply their Python functions on dataframes on NVIDIA GPUs with great flexibility while achieving the maximum performance. The feature’s idea of combining Python and just-in-time compilation applies beyond the scope of dataframe extract, transform, load (ETL), and has potentially many more use cases.

--

--

1

Published in RAPIDS AI

RAPIDS is a suite of software libraries for executing end-to-end data science & analytics pipelines entirely on GPUs.

Written by Jiqun Tu

Responses (1)

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

CUDA Neural Network Implementation (Part 1)

Paweł Luniak

Follow

--

Listen

Share

When you want to try out some neural network architecture, your method of choice will be probably to take some popular deep learning library (TensorFlow, pyTorch, etc.) and implement your network in Python. Well, it seems as the right way to do it but have you ever wondered what happens under the hood of these libraries? Let’s face the truth, the core of these libraries is not written in Python. The vast mojority of critical code is implemented in C/C++ language. What is more, if you desire to really speed up the computations, you need to take advantage of your GPU processor. And here, CUDA platform comes into play. In this post I’m going to present a very simple implementation of feedforward network using CUDA. In the next part I will focus more on ways we can optimize this simple implementation. I assume the reader is familiar with main neural networks concepts and with C++ language. Complete code is available at github.

Table of Contents

What is CUDACUDA Programming ModelImplementation PlanImplementationTrainingConclusion

What is CUDA

CUDA is a parallel computing platform intended for general-purpose computing on graphical processing units (GPUs). It stands for Compute Unified Device Architecutre and is developed by NVIDIA. But let’s start from the beginning. Parallel computing is a method of performing computations, where many operations are carried out simultaneously. GPU unit is an example of a device which is able to perform such computations. Originally GPUs were created in order to assist in the process of an image generation. Nevertheless, today they can be harnessed to perform many other computations which is called general-purpose computing on graphics processing units (GPGPU). In 2006 NVIDIA announced CUDA architecture along with C language dedicated for GPU. Very similar technology called OpenCL was published in 2009 and, unlike CUDA, is implemented in hardware produced by different companies, not only by NVIDIA.

GPU vs. CPU

One may ask, why do we even bother about some GPU if CPUs are so efficient today. They are, but in terms of sequential processing. In many applications we need to perform vast amount of exactly the same operations, e.g. add two very big vectors with 1M elements together. It turns out that we can have our result much faster using a lot of weak computing units, where every single unit has to take care just about 1000 vector elements, rather than having one powerful unit that has to compute all 1M results. “Strength in numbers” they say. While most CPUs have 4 or 8 cores, GPU can have even 3072 cores (NVIDIA Tesla M40) optimized for masivelly parallel computations. In the video below you can see really vivid explanation of difference between CPU and GPU.

CUDA Programming Model

One of the most important concepts in CUDA is kernel. Kernel is just a function that is executed in parallel by N different CUDA threads. When kernel is called we have to specify how many threads should execute our function. We distinguish between device code and host code. Device code is executed on GPU, and host code is executed on CPU. Let’s look at some example.

As you can see, kernel is defined with __global__ keyword which means that this function runs on the device and is called from the host code. We can call this function with addVectors<<<blocks_number, threads_in_block>>>(...), so in the example above we run 1 block of N CUDA threads. Inside our kernel function we can obtain index of current thread with threadIdx.x. In this case our thread block have just one dimension of N threads, but we can also execute multidimensional thread blocks. For instance, if we need to operate on matrices it would be much more convenient to run block of NxM threads and then obtain matrix row and column with col = threadIdx.x; row = threadIdx.y.

Thread Hierarchy

In a single block we can have up to 1024 threads. Therefore, we usually need more blocks in order to execute much more threads. Just as we can have multidimensional thread blocks, we can have multidimensional grid of blocks as well. For convenience we can use dim3 type to define either grid of blocks or block of threads. Let's refactor our kernel invocation code fragment to have 25x25 grid of blocks, and each of them have 32x32 threads.

In the figure below you can find visualisation of all this concept about grid of blocks and blocks of threads. As you can see, we have many blocks in a grid and every single block consist of number of threads. We can lay out these elements in 1D, 2D or 3D manner.

Memory Hierarchy

There is one more thing we have to cover to complete this quick CUDA programming introduction — memory. As you know, CPU has RAM memory for its disposition, as well as GPU has its own, separate RAM memory. These memories have different address spaces and CPU cannot easily access data that reside in GPU memory and vice-versa. Therefore, we need to manage transfer of data from CPU memory to GPU memory and the other way around. We also need to allocate host memory and device memory separately. So let’s make our code complete and make sure that vectors’ elements will be accessible from within a kernel, i.e. on a device (GPU).

cudaMalloc(...) allows us to allocate memory on device and cudaMemcpy(...) is used for data transfer between host and device. And that's it, our two vectors can now be correctly added and we can access this operation result in the host code. The memory we are using here is called global memory. It has the biggest capacity (several GB) and all CUDA threads have access to it, but it is also the slowest one. If you want your CUDA kernels to be fast, memory access performance is what you should really care about. Each thread block has shared memory visible only for its threads. It is much faster than global memory, but it has also much lower capacity (several dozen of kB). Every single thread is also equipped in private local memory that is the fastest one, but it is visible to a single thread.

With CUDA 6, NVIDIA introduced unified memory that is a pool of managed memory that can be shared between CPU and GPU. It is accessible for CPU and for GPU and really simplifies memory management when writing CUDA programs. To allocate space in unified memory we have to use cudaMallocManaged() function. If you want to know more about it, check Further Reading section.

Implementation Plan

We know already fundamentals of CUDA programming. We can use that knowledge to prepare a simple feedforward neural network implementation and harness GPU power for this purpose. Let’s say we want to be able to build a network shown in the figure 3. It is just linear layer followed by ReLU activation and one more linear layer followed by sigmoid activation.

What do we need more? Ah yeah — a cost function. Let’s say we would like our network to solve binary classification problem, therefore we will use binary cross-entropy function. In order to implement a whole neural network we will need following classes:

Implementation

We will go through implementation of every class listed above, but to keep this post on track we will skip some implementation details. Complete working code along with unit tests is available at github. Let’s start with Matrix class implementation.

Matrix Class

We need some data structure that will keep all the numbers and parameters — a matrix. The important thing here is that we want a matrix to be available both on host memory (for CPU) and on device memory (for GPU). Most operations will be performed on device but we might need e.g. to initialize a matrix on host, just because it will be easier. Of course we could implement everything on device, but we want to keep it simple. Matrix class will manage memory allocation and make data transfer between host and device easier. On listing 4 you can see a Matrix class header.

As you can see we are using smart pointers (std::shared_ptr) which will count references for us and deallocate memory when suitable (both on host and on device). In a while you will see how we will allocate device memory with smart pointer. The most important functions here are allocateMemory() (allocate memory on host and on device) and two functions for data transfer i.e. copyHostToDevice() and copyDeviceToHost(). A Shape is just a structure that keeps X and Y dimensions. allocateMemoryIfNotAllocated() function checks whether memory is already allocated and if not it will allocate memory for a matrix of a given shape. This will be useful when we won't know the matrix shape upfront. For convenience we overload subscript operator [] to have easy access to matrix values (from data_host). Let's look at functions performing memory allocation. In listings 5 and 6 you can see host memory allocation and device memory allocation using smart pointers.

As you can see in the listing 5 we have an ordinary memory allocation with new operator. We are passing pointer to an allocated memory to shared_ptr (1st argument). As smart pointer by default will call delete operator we need to pass also a second argument that will tell how to perform deallocation. We can put here a pointer to a function or, as we did here, enter a lambda expression with delete[] operator.

In case of device memory allocation we need to perform analogous operations, but on device, i.e. for GPU. Firstly we allocate memory using cudaMalloc(...) function, then we pass a pointer to allocated memory space to shared_ptr. Again we are passing lambda expression as the second argument but this time we are deallocating device memory, so we need to use cudaFree(...) function instead of delete[] operator.

Memory allocation is the most important and usefull thing when it comes to Matrix class. allocateMemory() function just call two presented above functions. When it comes to copyHostToDevice() and copyDeviceToHost() functions they just call cudaMemcpy(...) function in suitable direction, i.e. from device to host or the other way around.

Layers Classes

Every class that will implement any neural network layer has to perform forward and backward propagation. What is more we want NeuralNetwork class to treat every layer the same way. We don’t want to care about implementation details of layers classes, we just want to pass some data to every of them and get some results. All right, it sounds like we need polymorphism. We need an interface for all network layers — you can see it in the listing 7.

To sum this interface up, every layer is required to have forward(...) and backprop(...) functions. Each layer has also some name.

Sigmoid Layer

Both activation layers we are implementing are very simple. Sigmoid layer should compute sigmoid function for every matrix element in the forward pass. The sigmoid function is:

In the backward pass we need to use the chain rule which is foundation of backpropagation. We want to compute an error introduced by layer’s input Z. We will denote this error as dZ. According to the chain rule:

Where J is a cost function and dJ/dsigma is an error introduced by sigmoid layer — we obtain it as backprop(...) function argument and we denote it as dA.

Below you can see how SigmoidActivation class header looks like.

Note that we want to store layer’s input Z as well as its output A. We want also to store output of backpropagation from this layer, this is denoted here by dZ (because we are calculating error introduced by layer's input Z).

I am using here following convention: Z denotes output of a linear layer, A denotes output of activation layer. In general we can write operation performed by sigmoid function as A_n = sigma(Z_n), where Z_n = W_n A_(n-1) + b_n is the output from linear layer.

We start by implementing CUDA kernels for our sigmoid layer. In the listing 9 you can see forward pass kernel.

The implementation is rather straightforward. We calculate index for current thread that is executing the kernel, then we check if this index is within matrix bounds and compute sigmoid activation. Every CUDA thread compute a single output. What might be new here is a function with __device__ keyword. This is a function that can be called only on device, e.g. by our kernel function, and is executed on device. On the other hand __global__ functions can be called from host and are executed on device.

Backward pass logic is very similar to forward pass, the only difference is that this time we are implementing another equation. As we have kernels already implemented, we can call them in forward(...) and backprop(...) functions of SigmoidActivation class. Let's see how it is done.

In forward pass we store Z input, because we will need it during backpropagation step. Then we make sure that output matrix A has allocated space and a proper shape. After that we compute number of blocks we need, so that every of them contains 256 threads. After computations we return result matrix A.

There is nothing extraordinary in backprop(...) function if we compare it with forward(...) function. We just make sure that dZ matrix, that is an output of this function, is correctly allocated. Next we define number of threads and blocks and call a kernel. This pattern repeats in every layer implementation, therefore I won't list these functions for further layers. The most interesting things for us are actually kernels and this is what we will analyse hereafter. If you are interested in all details, please check the github repository.

ReLU Layer

ReLUActivation class has almost the same header as SigmoidActivation class, so we skip it. The main difference here is equation defining ReLU function:

Note that derivative of ReLU function is 1 when x > 0 and 0 otherwise. Therefore for backpropagation, using the chain rule, we obtain following formula to implement:

In the listing 13 you can find implementation of the forward pass kernel.

As we already know equations, implementing CUDA kernels is quite simple and in its logic very similar to sigmoid layer kernels. Within CUDA kernels we can use number of math built-in functions, one of them is fmaxf function. More on built-in functions you can find in CUDA Math API Documentation. Now it's time for backward pass implementation.

Again, we just implement equation presented above. There is one thing worth to notice in this place. We had to use if statement here in order to check whether Z input was greater or lower than 0. This cause that some threads will execute different set of instructions than other threads. This hugely affects the kernel performance and we should in general avoid if statements in our kernels as much as possible. On the other hand the first if that checks matrix bounds is not so adverse, because most threads execute the same code anyway (most threads will have index within matrix bound). This is called thread divergence and I will write more about it in the part 2 of this post.

Linear Layer

LinearLayer class is more interesting than classes of activation functions because it has parameters W and b, so a bit more happens here. In particular we need to implement gradient descent to update these parameters during back propagation. As for previous layers we will start with equations. Linear layer in a forward pass should implement following equation:

Where W is weights matrix, b is bias vector and A is input to this layer. Note that this time we are computing Z, not A as in case of activation layers. Furthermore we need three derivatives this time. One to find out what error was introduced by input A, and this one should be passed to preceding layer during back propagation. We need also to know what error was introduced by W and b to be able to update these parameters accordingly using gradient descent.

Where dZ^i is i’th column of dZ (i’th input in a batch) and m is size of a batch. Below you can see how the header of LinearLayer class looks like.

As a quick overview let’s skim over functions of this class. Of course we have forward(...) and backprop(...) functions here and additionally we have some set of getters. We have two initialization methods here, i.e. for bias initialization and for weights initialization. We will initialize bias vector all with zeros and we will initialize weights matrix with values from normal distribution with 0 mean and standard deviation equal to 1. These will be additionally multiplied by weights_init_threshold value. If you are interested in mentioned initialization methods you can find details in the github repository. I guess that computeAndStore and update functions are self-explanatory. These are just helper functions that will call relevant kernel and these are used in forward(...) and backprop(...) functions. Let's go straight to the most interesting part that are CUDA kernels.

This time we don’t have to compute a function value for every matrix element independently as in case of activation functions. Now we need to compute matrices product. We should avoid synchronization of threads if it is possible, bacause it slows down our kernel. In order to omit the synchronization two threads cannot write to the same localization because otherwise we will have race conditions. On the other hand our threads can read from the same location simultaneously, i.e. values from multiplied matrices, because we know that these won’t change during computations. We can do this easily by asking every thread to calculate a single element of an output matrix. You can see it in the figure 4. Every single thread will compute a single pink dot. Every pink dot is a dot product of row from matrix A and a column from matrix B.

In the listing 16 you can find how forward propagation is implemented as a CUDA kernel.

Until now, we were creating 1D threads grid but here we create 2D threads grid. It makes things a bit easier, because by Y thread’s index we can get a row of the result matrix and by X thread’s index we get a column of the result matrix. Then we simply multiply each row element of matrix W with each column element of matrix A. Finally we add a bias to our result. That’s it. We computed output of a linear layer forward pass, i.e. Z matrix. Now we need to implement a whole backpropagation step. First of all we need to compute dA that will be passed to preceding layer.

This kernel is quite similar to this for forward pass. The only difference is that we need W matrix to be transposed. Instead of making separate kernel for transposition we can simply multiply each W column by dZ columns. It will be equivalent of computing transpose(W)*dZ. Next step is to update linear layer's weights accordingly to dZ which is presented in the listing 18.

We apply similar trick here to pretend that A matrix is transposed. The final step in this kernel is updating weights matrix. We are using the simplest form of gradient descent here and just subtract gradient value multiplied by learning rate from current weights matrix. The last step during backpropagation in our linear layer is performing bias vector update.

Bias update kernel is very simple and we simply apply gradient descent rule. What is iteresting in this kernel is usage of atomicAdd(...) function. When we implement bias update this way, multiple threads will write to the same memory location, i.e. bias vector elements. We have to make sure that we won't have race conditions here, therefore we call atomic operation that guarantee that another thread will have access to memory location when current thread complete his addition operation. Nevertheless, using atomic operations in CUDA kernels is undesirable because it harms kernel performance. Instead of quickly performing a lot of operations, some threads need to wait for their turn. Of course, sometimes we just have to use atomic operations and CUDA provides some set of such.

All right. Implementation of all neural network layers is ready. We still need a cost function.

Binary Cross-Entropy

We have decided to use binary cross-entropy cost function, so what do we need exactly? Well, just a function that computes a cost and a function that returns gradient accordingly to network predictions and our target values. Binary cross-entropy is defined with following equation:

And by calculating its derivative we compute gradient as:

Where by y-hat we denote predicted values and by y the ground truth values. The header of BCECost class is straightforward.

Nothing extraordinary happens in BCE cost calculation kernel. Note that we use again built-in math function, this time logf.

And below is gradient computation kernel.

So far we have implemented all fundamental building blocks of our neural network. We just need one more class — NeuralNetwork, that will keep them all together.

NeuralNetwork Class

This class is responsible for managing all network components and also to communicate layers with each other during forward and during backward passes. Let’s look at its header declaration.

Our NeuralNetwork class keeps all layers in layers vector. We can add new layers using addLayer(...) function. The class holds also cost function object. We could probably make this more generic to allow user to put his own cost function, but let's leave it that way for now. The most important functions here are forward(...) and backprop(...). These function are just passing output from one layer to the next one (forward pass), or to the previous one in case of back propagation. In the listing 23 you can find code of the forward(...) function.

As you can see, all we do here is to iterate over every layer and pass output from one layer to another. Of course, to the first layer in the vector we pass network input X. The backprop(...) function is very similar, but we have to use reversed iterator to go through the layers vector from the last layer to the first one.

Back propagation function takes as its arguments predicted and target values, computes gradient of BCE cost and then passes errors between all network layers calling their backprop(...) functions. A new thing here is cudaDeviceSynchronize() call. This function waits untill all CUDA threads end its job. It is similar to join() function when we are working with traditional threads. We need this function because we want to get cost value from host code during training time. We have to be certain that all computations are over, otherwise we can obtain some pseudorandom values.

Training

We have implemented everything we need to build neural network based on linear layers, ReLU and sigmoid activations and binary cross-entropy cost. Let’s create a neural network that was described at the beginning and train it to predict some values. I created CoordinatesDataset class to check whether our network is able to learn something. It just draws random coordinates in 2D space. We want to predict whether a point lies within 1st or 3rd quadrant or whether it lies within 2nd or 4th quadrant.

It is time to add some layers to our network and try to train it so it will predict to which class a given point in 2D space belongs.

At the beginning we create a dataset with 21 batches, every one containing 100 data points. We will use 20 batches for training and the last one as our test set to check the final accuracy score. Then we populate our neural network with some layers. First layer takes batch of points, so we have 2 inputs here and we would like to have 30 neurons in the hidden layer. Then we add ReLU activation and define new linear layer with 30 neurons that will output just a single value that will be activated with sigmoid function. The next part is a training. We are going to train the network over 1000 epochs and print the cost after every 100 epochs. After training we can check the accuracy score of our neural network. computeAccuracy(...) function just counts number of correctly predicted values and divides it by the size of output vector. You can find the output in listing 25.

As you can see the neural network converges over number of epochs. The convergence is a little bit slow which is probably a result of using simple gradient descent. After 1000 epochs the network scores 93% on accuracy. With such simple dataset we can easily make it 100% with more epochs. Finally it seems that our CUDA implementation of neural network is working correctly.

Conclusion

In this a little bit lenghty post we have implemented simple neural network using CUDA technology and performed the crucial operations on GPU processor. GPU is great for massively parallel operations such as matrix operations. However, originally GPUs were not designed for general-purpose computations including neural networks implementation. This is one of reasons why hardware companies are still working on much more suitable processor architectures to get as high performance as possible on such applications, though it’s a topic for another post.

An important thing is that writing CUDA kernels is not as easy as it might seem at first glance. We have to be cautious about different conditions and restrictions to make computations correct but also to not harm kernels performance. What you have seen in this post is a very simple implementation and it can be improved. This is definitely not the implementation you would find if you take a look at popular deep learning libraries repositories. Nevertheless, I hope it shows that there is quite a lot going on under the hood of python deep learning libraries. I am going to write a second part of this post, where I will focus more on the performance issues and possible improvements of this implementation.

Further Reading

GitHub Repository — Here you can find complete code for the project presented in this post along with unit tests.An Even Easier Introduction to CUDA — An excelent introduction to CUDA. It uses unified memory instead of manual transfer of data between host and device.Unified Memory in CUDA 6 — More information on Unified Memory introduced in CUDA 6.CUDA Thread Execution Model — An extensive information on CUDA thread execution model. You can find here more about thread synchronization, scheduling and divergence.CUDA Toolkit Documentation — Everything you might need when working with CUDA.

Originally published at luniak.io.

--

--

Written by Paweł Luniak

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

How to squeeze every ounce of performance out of the robot 🤖?

Mayur H

Follow

--

Listen

Share

My Journey with OpenACC and CUDA in Robotics .

Hey there! If you’re into robotics like me, you’ve probably heard of OpenACC and CUDA. If not then buckle up because If you’re serious about robotics, I can’t stress enough how important it is to get comfortable with hardware optimization tools. They’ve become the daily needs in my toolkit for squeezing every ounce of performance out of the robots I work on and I’m sure they can do the same for you.

Remember, every millisecond counts in robotics. Whether you’re making a simple robot arm or explore Mars, these tools can be the difference between a good robot and a great one. So let me explain why and how far you can push your robotic performance!

Why I Care About Hardware Optimization?

In the world of robotics, getting the most out of your hardware isn’t just nice to have — it’s essential. It’s the difference between a robot that sluggishly goes through the motions and one that zips through tasks with precision. I’ve seen firsthand how optimized robots work faster, last longer on a single charge, and handle more complex jobs without breaking a sweat (or overheating, in robot terms).

Whether you’re tinkering with a vacuum bot or programming industrial arms, trust me, hardware optimization is an absolute necessity. It’s not just about performance; it’s about cost-effectiveness too. Who doesn’t want their robots to work better and cost less, right?

Why I Started Using OpenACC and CUDA

When I first started building robots, I hit a wall with performance. My algorithms were solid, but my robots were sluggish. That’s when I discovered the world of GPU acceleration and parallel computing.

OpenACC and CUDA are both tools for parallelizing code, but they work differently:

OpenACC:

I started with OpenACC because it was easier to grasp. Here is a simple example of OpenACC to give you an intuition::

This is an OpenACC directive for parallelizing a loop. This simple directive parallelized vector addition, giving a significant speed boost in sensor data processing. OpenACC is a high-level, directive-based approach to parallel computing that can be used with C, C++, and Fortran. The #pragma acc parallel loop directive tells the compiler to parallelize the following loop, distributing the work across available computing resources (typically GPU cores).

CUDA: When I Needed More Power

As my projects got more complex, I moved to CUDA. It’s more challenging but offers finer control:

Here’s a basic CUDA kernel used for the same vector addition:

This is a CUDA kernel function and its invocation.

CUDA is NVIDIA’s parallel computing platform and programming model for their GPUs. The __global__ keyword indicates that this is a kernel function that runs on the GPU and can be called from the CPU.

The kernel function vectorAdd performs element-wise addition of two vectors a and b, storing the result in result. The if (i < n) check ensures that the thread doesn't access memory beyond the array bounds.

The last line shows how the kernel is launched from the main function, using CUDA’s triple angle bracket syntax to specify the number of blocks and threads per block.

This level of control allows to optimize a computer vision algorithm for real-time object detection on a onboard compute.

Both snippets are implementing the same operation: parallel vector addition. The OpenACC version is more abstract and relies on the compiler to handle the parallelization details, while the CUDA version gives the programmer (you ) more explicit control over the parallelization strategy.

Technical Challenges I Faced and you probably will too !

1. Memory Management:

2. Optimization Techniques:

3. Debugging:

Real-World Applications

In my projects, I’ve used OpenACC for simpler tasks like basic sensor data processing. It’s great for those “set it and forget it” kind of jobs where you don’t need to micromanage every computation.

But when I’ve worked on more demanding tasks like real-time SLAM or complex path planning, that’s where CUDA really shines. I remember spending days optimizing a vision system for a high-speed robot, and CUDA’s fine-grained control made all the difference in achieving the performance we needed.

Choosing Between OpenACC and CUDA

Choosing between OpenACC and CUDA often feels like deciding between an automatic and a manual car. OpenACC is easier to drive, but CUDA gives you that extra control when you need it.

I’ve found that for quick projects or when working with a team that’s not deep into GPU programming, OpenACC is a lifesaver. But for those critical, performance-intensive applications, rolling up my sleeves and diving into CUDA has always paid off.

Why Learn Both?

Knowing both is like being skilled in driving both manual and automatic car. They exercise different skills and are useful in different scenarios. OpenACC gets you up and running quickly, while CUDA lets you fine-tune performance when every millisecond counts.

In my experience, understanding both has made me a more versatile roboticist. I can quickly prototype with OpenACC and then optimize critical parts with CUDA when needed.

Tips for Beginners

🚀 Bonus

Below are some of the interview questions for Robotics/Computer Vision roles which I have been asked in internships/ full time job interviews

These questions test not only the candidate’s knowledge of OpenACC and CUDA but also their ability to apply this knowledge to complex robotics and computer vision problems. They require a deep understanding of parallel computing, algorithm design, and hardware optimization in the context of real-world robotic applications.

Happy Learning 🙌🏽

--

--

Written by Mayur H

👨🏽💻Staff Computer Vision & Robotics Engineer 🤖 Building Embodied Intelligence 📍 London 🇬🇧

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams