Sign up

Sign in

Sign up

Sign in

Analytics Vidhya

Home

Newsletter

About

Analytics Vidhya is a community of Generative AI and Data Science professionals. We are building…

Sparse Matrix-Vector Multiplication with CUDA

Georgii Evtushenko

Follow

Analytics Vidhya

--

1

Listen

Share

Introduction

Standard methods of differential equations discretization usually lead to systems of linear equations. General feature of produced systems is that the number of entries in each equation depends on local topological features of the discretization. Thus, the matrices generated by these systems contain a lot of zeroes (fig. 1). It’s possible to take advantage of knowledge about position of zeroes by storing matrices in special data structures. The abstract data type for these structures is called sparse matrix. While I was reading about yet another matrix format, I decided to actualize the comparison of performances of different matrix formats. This post provides an review of efficiency for basic sparse matrix data structures in the context of sparse matrix-vector multiplication (SpMV) on GPU.

Data Structures for Sparse Matrices

In general, SpMV performance is limited by memory bandwidth. The storage formats, which are used for the sparse matrices define SpMV algorithms. Each of these algorithms has its own granularity, which impacts performance. The primary distinction among sparse matrix representations is the sparsity pattern, or the structure of the non-zero entries, for which they are best suited. However, I’ll start with general sparse matrix formats.

To access the efficiency of SpMV on different sparse matrix formats, I’ve collected performance data on general matrices from Florida Sparse Matrix Collection. All of the experiments were run on a system with NVIDIA RTX 2080 GPU paired with an Intel Core i7–7700k CPU. Each of the measurements is an average (arithmetic mean) over 30 trials. Before measuring performance, both CPU and GPU frequency were fixed.The speedup was computed by dividing single thread CSR SpMV execution time by GPU one.

CSR

The Compressed Sparse Row (CSR) format is a general sparse matrix format. CSR format consists of three arrays: row_ptr, columns of non-zeroes, and matrix values (fig. 2). The non-zero values of the row are stored consequentially in an one-dimensional values array. The row_ptr array is used to divide values array into separate rows. Its size is equal to n_rows + 1. The last entry in row_ptr stores a number of non-zeroes (nnz) in the matrix. That allows fast querying of non-zeroes number in a particular row (row_ptr[row + 1] − row_ptr[row]). For each non-zero value column index is stored in columns array.

Let’s assume for simplicity that there are four threads in each CUDA thread block. General CSR SpMV implementation works at the granularity of threads per row (fig. 3). Hence, the matrix in figure 2 is processed by three thread blocks. This implementation is usually referenced as CSR-Scalar (list. 1).

Presented implementation of CSR SpMV algorithm on GPU is usually considered very inefficient. The reasons of inefficiency are load balancing, thread divergence, and memory access pattern. As shown in figure3, only half of the block threads has non-zeroes to process. Thus, a single dense row can arbitrarily delay the execution while all the other cores are idle. Moreover, as shown in figure 3, adjacent threads access matrixvalues in a strided way. When concurrent threads simultaneously access memory addresses that are far apart in physical memory, then there is no chance for the hardware to combine the accesses. Performance resultsfor naive CSR-Scalar implementation are presented in table 1.

The speedup distribution is shown in figures below. To answer the question how naive described implementation really is I’ve compared it with the NVIDIA CUDA Sparse Matrix library (cuSPARSE) CSR implementation (tab. 2), which has a better average speedup.

These results show that there is a room for optimization of CSR SpMV. The first possible optimization is to assign warp per row instead of thread. This algorithm (list. 3) is called CSR-Vector. The vector kernel accesses indices and data contiguously (fig. 4), and therefore overcomes the principal deficiency ofthe scalar approach. Unlike the previous CSR implementation, which uses one thread per matrix row, this optimization requires coordination among threads within the same warp.

In the case of CSR-Vector reduction might be implemented using warp-level primitives (list. 2). In that case, the data exchange is performed between registers and more efficient than going through shared memory, which requires a load, a store, and an extra register to hold the address.

CSR-Vector has better speedup (tab. 4) and speedup distribution than CSR-Scalar (for both float and double matrices) and cuSPARSE implementation (for float matrices).

However, CSR-Scalar outperforms CSR-Vector on about 33% of float matrices with 10000 nnz lower limit and on 40% of float matrices with 100000 nnz lower limit. On that matrices, CSR shows average speedup equal to 8.57 while CSR-Vector only 4.80.

To discover further improvements of CSR SpMV implementation, we need to consider the first matrix part from figure 2. In the first four rows of the matrix, there is only one non-zero value per row. In that case all threads of warp except first are idle. In this case, it’s possible for naive CSR SpMV implementation to outperform vector implementation. There is an SpMV algorithm for the CSR matrix format that doesn’t depend on nnz/row ratio. The CSR-Adaptive changes it’s behavior depending on the nnz in each row (list. 4). After selecting non-zeroes per block value, additional array (row blocks) for storing rows of block is constructed. If some rows contain small nnz, they’ll be gathered into one block. Then CUDA threads block is assigned to each block of rows. The case of multiple rows in one block of rows is called CSR-Stream. If there is only one row in block of rows, the CSR-Vector will be called. If this row exceeds nnz_per_wg than CSR-VectorL variant will be used. The main difference between CSR-Vector and CSR-VectorL is that CSR-VectorL allows executing multiple CSR-VectorL on one row and then reducing the results by using atomic operations.

The CSR-Vector and CSR-VectorL parts are quite similar, so I won’t include listing here. Figure 5 illustrates memory access pattern of the CSR-Stream part. It stores partial sums in shared memory of GPU and then reduces them. The partial results in cache in figure 5 are calculated with x filled with 1. Thesource code of CSR-Stream is presented in listing 5.

On the discussed set of matrices, where CSR outperformed CSR-Vector, CSR-Adaptive shows better speedup. CSR-Adaptive outperforms CSR-Scalar on those 291 matrices. Although CSR-Adaptive might be outperformed by CSR-Vector on some long-row matrices, it has better speedup in average (tab. 4). The main advantage of CSR-Adaptive is that you won’t need to change the code that generates a matrix if your code already uses CSR. The matrix formats presented below don’t have this quality.

ELL

The problem of noncoalesced memory accesses of CSR can be addressed by applying data padding and transposition on the sparse matrix data (fig. 6). The Ellpack-Itpack (ELL) sparse matrix format assumes that each row contains at most elements in rows elements and elements in rows is small. All rows are zero-padded to that value. Unlike CSR, the rows pointers array is of no need. ELL is most efficient when the maximum number of nonzeros per row does not substantially differ from the average.

Kernel for ELL matrix format is presented in the listing 6. With element padding of the ELL format, it’s easy to get the next row’s element position by simply adding the number of rows in the matrix. The padding also fixes the number of iteration for each thread, so there is no control flow divergence in warps. Elimination of control flow divergence and enabling of memory coalescing allow ELL SpMV kernel to outperform CSR-Scalar implementation on many matrices (tab. 5).

The obvious disadvantage of ELL format consists of padding itself. In the case of a matrix with a few long rows, ELL format will result in an excessive number of padded elements. There are a lot of matrices in Florida Collection, that couldn’t fit into 8GB of my GPU because of ELL’s padding. In some cases, it leads to a situation where CSR-Scalar outperforms ELL implementation. To eliminate this issue, it’s possible to remove long rows’ extra nnz from ELL matrix into the different matrix. It is important to note that extracted matrix would have an unordered scheme. Many rows will likely be missing from that scheme, so CSR using would be inefficient. One of the formats that could handle that case is COO.

COO

The coordinate (COO) matrix format is the simplest one. For each NZ it stores it’s column and row indices. Therefore, COO doesn’t map elements in rows. That leads us to the necessity of atomic operations in COO kernel (list 7).

COO SpMV implementation works at the granularity of threads per element (7). Atomic updates to the result vector reduce performance. The wider rows in COO format, the more serialized SpMV is. This fact can be noticed in figure 7. To improve the performance of this format, it’s possible to slice the matrix info chunks with the rows count that fits into shared memory.

Matrix format that uses shared memory to improve atomic operations performance in COO SpMV is called Sliced COO (SCOO). To reduce shared memory bank conflicts, SCOO allows multiple lanes in the shared memory for updating the intermediate results of a single row. Reducing slice size increases the size of lane, and thus more shared memory lanes are available.

Hybrid

It’s possible to use ELL matrix format on the regular part of the matrix and COO on the elements removed from extra-long rows. This scheme significantly reduces the number of padded elements in ELL format. Thisapproach is often called as hybrid. There are different options for combining the results of ELL and COO SpMV. In this post I use atomic case (list. 8).

Althought the average performance results (tab. 8) are quite close to CSR-Adaptive SpMV, Hybrid format requires extra actions for the splitting matrix, which might require rewriting of a matrix calculation code base.

Conclusion

To conclude this post, I would like to show you some misleading results. I’ve selected some matrices (tab. 9) to show the obvious fact that there is no universal matrix format. The leader changes even after data type change. In my next post, I’m going to focus on block matrix formats generated by real applications. Source code and pdf version of this post are available in github.

--

--

1

Published in Analytics Vidhya

Analytics Vidhya is a community of Generative AI and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Written by Georgii Evtushenko

C++ Software Developer with experience in high-performance computing.

Responses (1)

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

What I wish I knew when I started programming CUDA (Part 2)

Robby

Follow

--

Listen

Share

This is the second part of my CUDA series. In the first post I showed a few recourses that are helpful, narrowing down the vast field of CUDA trainings and teaching videos to the essential ones. I also tried to decipher the concept of streaming multiprocessors, blocks and threads, and how they relate to each other. Finally, I talked about warps and why they are useful. In this post, firstly I want to talk a bit on kernel launch configurations. Then I want to expand on warps and how to (not) use them appropriately, and what to look out for in data structures. Finally, I want to give a hint on which high level libraries help you design code faster.

Kernel launch configurations and limits: It is known that when launching kernels, the number of blocks and threads have to be chosen. Those are called the kernel launch configurations. While the kernel launch configurations are individual to each problem, they should however remain a multitude of 32, the size of a warp. Moreover, they underlie certain limits. At this point, according to the thread hierarchy chapter of the CUDA programming guide, it is 1024 threads per block on current GPUs. There is furthermore a limit on the blocks that can be evoked, which is subject to the compute capability of the device. The exact specs can be taken from the specs of the programming guide. Currently, in the x-direction, they are limited to 2³¹-1, in the y- and z-direction they are limited by 65535 blocks.

Flattened arrays and memory accesses: In the previous post I have explained how CUDA uses warps to simultaneously fetch data from memory and tries to process it simultaneously. This has a few implications on how to structure the data in use. Technically, CUDA provides the user with means to design multidimensional algorithms by embedding three-dimensional data structures in its syntax, such as the dim3-structure to specify grids and blocks, as shown in the example below.

This code simply calls 32 x 32 blocks, and each one of them would have 128 x 128 threads. Indexing within this multidimensional setting goes via the next code snippet:

It is easy to see that multidimensional arrays could be indexed very easily like this. They can be created just like C-style multidimensional arrays, realized via pointers to pointers. The catch however comes when warps get into the equation. Imagine for example when you want to index a 3x3 sized block of pixels of an image, like depicted in the image to the left.

In the block to the left, the indexed pixels are the orange ones, whereas the blue ones are the normal surrounding ones. The code could be looking as in the example below. In that case, each time the GPU would fetch not only fetch the 9 elements of the indexed pixels, but instead fetches the data in batches, and then only uses one single element per batch to copy it into a single variable. To make matters worse, although accessing a two-dimensional pointer, the GPU memory is still more of a linear array, introducing strided memory accesses that Mark Harris is talking about in this post.

An initial remedy would be to flatten out the image. Rather than having a pointer to a pointer, the data could be stored in a flat array. Knowing the dimensions of the image would be enough to determine row and column, and the unnecessary memory fetches at the edge of the two-dimensional solution would be a matter of the past. In order to alleviate the strided memory accesses, depending on the problem different solutions are possible. What could help here is the uses shared memory, a technique Mark Harris shows in order to transpose a matrix efficiently. However, it is up to the engineer how to solve this. The key takeaway is that memory accesses should be in continuous ways in flat structures, and that strided memory accesses should be avoided in order to make best use of warps.

Data structures: Expanding on the idea above of flat structures, data structures such as structs or classes should be handled with extra care. Although a bit of a superficial example, the following code has two kernels implementing the same purpose, namely for each point of an array of points compute the sum of its x, y and z coordinates. In expectation, the second kernel using the arrays should be faster than the first kernel. An explanantion follows below.

As can be seen, the first kernel, sumPoint(), takes in a point data structure, with the components distributed. The second kernel however receives three arrays, each one carries the individual component. Thus, they appear different in memory, and memory is fetched differently, as shown below. Because the second kernel, sumPointArray(), does parallelize fetching of the individual components, it uses the way a warp and SIMT work, simply better.

Thrust and supported libraries: Writing CUDA code can be an arduous process, because the normal interface serves a C-style memory management. In order to alleviate this and reduce problems arising from forgotten pointers, C++ supports the concept of smart pointers since the C++ 11 standard. The concept is simple: Create a wrapper class for the pointer, and when the object goes out of scope, free the memory from the constructor. Thrust is a high-level library support that behavior on CUDA. The interface is written in the C++ standard library (STL) vector style, allocating memory on the GPU and automatically transferring the data on it. It furthermore supports operations such as resize(), push_back() and normal indexing, and some high-level algorithms such as sorting. This library is very good for fast prototyping as well as more legible code, and I personally have not seen significant performance drops, if none at all. The documentation can be found here. Furthermore, CUDA supports other libraries, which could prove useful in areas such as signal processing or image processing. They can be found here.

This concludes my two-post series about CUDA up to now. I may add one or another chapter later, but at the moment I will keep it as it is. If you find mistakes or have questions please do not hesitate to ask me. I hope you enjoyed my articles and learned something from them, and best of luck on your projects involving GPU programming.

--

--

Written by Robby

Electrical and software engineer from Germany. I like to travel and get to know cultures, learn and explore life.

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

Update Reduce Operation

Lawliet

Follow

--

Listen

Share

In my last post, I talked little about the Reduce Operation (actually sum operation in my case). I will compare different strategies of Reduce Operation.

First, recall the raw result after my cuda kernel finished as shown in following picture:

Of course there are lots of libraries (e.g. cudpp, cub, etc.) that can do Reduce Operation (e.g. sum, minus, multiply, min, max, etc..). However, lack of them can perform in batch. Also, for each of these libraries, we need to manage several abstract pointers such as handles, plans, configs and so on. When programming with multi-GPUs, managing all these stuffs will become very trivial. Not to mention these libraries are not necessarily faster than cublas. Therefore, as cublas is indispensable in DL, why not just use cublas to finish this Reduce Op.

Actually, all the algorithms I will compare here are based on matrix-matrix or matrix-vector multiplication.

Since all the data in cuda is stored in 1-D layout, it’s very convenient to reorganize data depending on situations. All you need is a pointer to memory and how to interpret data pointed. For example, from the memory aspect, the raw γ in the Fig 1 is actually stored in this way:

each γ_i,j here is a vector. We take one row out and reorganize it as a matrix:

In cublas’s eyes, it regards all data in column major, so one row of raw γ is actually looks like right layout in Fig 2. But we want to use the left matrix in Fig 2 times with a 1s vector to get one row of final γ:

Fortunately, cublas can do matrix transposition before multiplication, so we don’t need to worry about that. On the other side, cublas doesn’t have API being able to do batch matrix-vector multiplication. Only batch matrix-matrix multiplication is supported. So we have to regard the 1s vector as a one column matrix. The code is as follow:

cublasSgemmBatched(cublasHandle,                    CUBLAS_OP_T,    // the raw γ needs to be                                    // transposited as Fig. 4                   CUBLAS_OP_N,    // the 1s matrix doesn't needs to                                     // be transposited                   channelCount,   // the number of rows of raw γ                                    // after tranposition                   1,              // the number of columns of 1s                    imgSize,        // the number of columns of raw γ                                    // after transposition                   &alpha,         // alpha = 1                   rawGamma,       // pointer to raw γ (Fig. 2)                    imgSize,        // the number of rows of raw γ                                    // before transposition                   onesMatrix,     // pointer to 1s matrix                   imgSize,        // the number of row of 1s                   &beta,          // beta = 0                   gradientGamma,  // final result of γ                   channelCount,   // the number of raw of γ                   channelCount)   // batch size

The above function will repeatedly do the matrix-matrix multiplication in Fig. 4 channelCount times.

2. For-loop Algo. — Don’t use batch

At beginning, I thought the batch API must be efficient, but the speed is not as I expected. In my neural network, I have totally 6 convolution layers, 6 GDN layers and 6 down/up -sample layers. By using 1st Algo., it takes more than 2 seconds for 4 images in one iteration. I’m so impatient that can’t help trying other methods. Then I decide to use for-loop instead of batch.

It’s actually similar to the first algorithm, but changes batch to for loop.

oneBatchSize = imageSize * channelCount;for(int j=0; j<channelCount; j++) {     cublasSgemv(cublasHandle,                  CUBLAS_OP_T,      // the raw γ needs to be                                    // transposited as Fig 4                 imageSize,        // the number of rows of raw γ                                    // before tranposition                 channelCount,     // the number of columns of raw γ                                    // before transposition                 &alpha,           // alpha = 1                 rawGamma+ j*oneBatchSize, //choose one column of                                            // raw γ (Fig 4.)                 imageSize,        // rows of 1s vector                 onesVector,       // pointer to 1s vector                  1,                // stride of 1s vector                 &beta,            // beta = 0                 gradientGamma + j*channelCount, // pointer to one                                                  // column of final                                                   // result of γ                 1));               //stride of result }

The advantage of this method is that we don’t need to use matrix-matrix multiplication but matrix-vector. If you’re interested in the cuda matrix-matrix multiplication, you can search the algorithm on Nvidia’s official document. I can only say that matrix-matrix multiplication algorithm is more inefficient than matrix-vector multiplication. Because matrix-matrix multiplication is sophisticated, which involves a lot of blocking and share memory stuffs. However, matrix-vector is relatively easy. Although our task is matrix-vector multiplication, as mentioned in last section, cublas doesn’t have batch matrix-vector. So we have to convert the problem to matrix-matrix.

Now the experiment shows that for-loop matrix-vector multiplication is faster than batch matrix-matrix multiplication. For the same size of input and same configuration of net, the computation time is now 800ms that is muuuch faster.

3. Super fast — Just need one matrix-vector multiplication

I know smart as you must notice this method if you understand the 2nd method. If you stare at the raw γ matrix for some time then close your eyes, the idea will come out to your minds.

Like I said, pointer is one of the greatest and loveliest feature in C++ because of it’s flexibility. Without any extra manipulation on data, all you need is just changing angles to look at the memory and start imagining.

Therefore, all you need to do is just one matrix-vector multiplication. I think the improvement is self-explained. And now the computation time is 200ms. Maybe you don’t understand what 200ms means. Once, I measured the the time spent on all convolution layers because I want to locate the bottleneck of my net. It’s 100ms totally. So, that means my GDN implementation is as fast as convolution layer implemented by Nvidia’s cudnn that is extremely optimized. That’s a huge jump.

Again I know there must still be some space to improve, but I cannot figure out how so far. Now I can train 1000 images in just one day which is much faster than my expectation.

For all these improvements, there is no magic and fancy theory. However, by implementing all these different algorithms, a lot of non-intuitive experiences are achieved. Therefore, when analyzing and optimizing your own cuda kernel, don’t just rely on your past experience about CPU programming. Doing all these experiments is very necessary. Usually simple changes will make difference.

--

--

Written by Lawliet

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

Mayin Dev

Follow

--

Listen

Share

I implemented a kernel that performs a single-pass scan (proposed in the book Programming Massively Parallel Processors):

Harnessing the Power of Parallel Processing with CUDA and GPUs

Parallel processing is revolutionizing the way we approach computationally intensive tasks. By breaking down problems into smaller, independent parts that can be solved simultaneously, we can achieve significant speedups compared to traditional sequential processing. CUDA (Compute Unified Device Architecture), developed by NVIDIA, provides a powerful framework for leveraging the parallel processing capabilities of GPUs (Graphics Processing Units) to accelerate a wide range of applications, from scientific simulations to machine learning algorithms. This article delves into the intricacies of CUDA programming and explores its advantages in the realm of parallel processing.

Understanding the Fundamentals of CUDA Programming

CUDA allows programmers to write code that runs on the GPU, taking advantage of its many cores. The GPU is organized into a hierarchy of blocks and threads, each performing a portion of the computation. Programmers write kernel functions, which are executed by these threads in parallel. Effective CUDA programming requires careful consideration of memory management, data transfer between the CPU and GPU, and efficient thread organization to maximize performance. Understanding the nuances of thread synchronization and data sharing is crucial for avoiding bottlenecks and achieving optimal speedups. Properly utilizing shared memory, a faster memory space within each multiprocessor, can significantly improve performance.

CUDA’s Thread Hierarchy: Blocks and Grids

CUDA’s parallel execution model is based on a hierarchical structure. Kernels are launched as grids of blocks, where each block contains a number of threads. This structure allows for a high degree of parallelism, enabling the execution of thousands or even millions of threads simultaneously. Understanding how to effectively organize threads into blocks and grids is essential for optimal performance, as it directly impacts memory access patterns and overall efficiency. Improper organization can lead to significant performance limitations.

Memory Management in CUDA

Efficient memory management is critical for high-performance CUDA programs. Data needs to be transferred between the CPU’s main memory and the GPU’s memory, a process that can be time-consuming. Strategies for minimizing data transfers and effectively utilizing different memory spaces on the GPU, such as global, shared, and constant memory, are essential for optimizing performance. Understanding the memory access patterns of your threads is also crucial to avoid memory conflicts and bottlenecks. The choice of memory type greatly influences the speed of access and the overall efficiency of your CUDA program.

Comparing CPU and GPU Parallel Processing

While CPUs excel at complex tasks and sequential processing, GPUs are optimized for massively parallel computations. This difference makes them ideal for different types of problems. CPUs generally have fewer, more powerful cores, while GPUs have many smaller, simpler cores. This architectural difference leads to significant performance variations across different applications.

Feature CPU GPU Number of Cores Few, powerful cores Many, smaller cores Clock Speed Higher Lower Memory Bandwidth Generally lower Generally higher Best Suited For Complex, sequential tasks Massively parallel computations

Real-World Applications of CUDA

CUDA’s versatility extends across various domains. From accelerating scientific simulations and image processing to powering machine learning algorithms and enabling real-time rendering in video games, CUDA’s impact is undeniable. The ability to offload computationally intensive tasks to the GPU frees up the CPU for other operations, resulting in substantial performance improvements. Many libraries and frameworks are built upon CUDA, simplifying the development process and providing access to optimized algorithms.

Advanced CUDA Techniques

Beyond the basics, mastering advanced techniques unlocks the full potential of CUDA. Optimizing memory access patterns, understanding and utilizing shared memory efficiently, and employing techniques like cooperative groups and warp-level programming can significantly enhance performance. Careful consideration of data structures and algorithms is critical to achieve optimal results. For instance, understanding how to effectively manage data dependencies within parallel kernels is crucial for preventing performance issues. This often involves careful consideration of thread synchronization mechanisms.

“Effective CUDA programming requires a deep understanding of both hardware architecture and parallel programming paradigms.”

For a deeper dive into optimizing kernel performance, consider this resource: Extending a Single-Pass Scan Kernel for Independent Row-wise Scan in CUDA. This blog post provides valuable insights into advanced techniques.

Conclusion

CUDA empowers developers to harness the immense parallel processing power of GPUs, leading to significant speedups in computationally intensive applications. By understanding the fundamentals of CUDA programming, memory management, and thread organization, developers can unlock the full potential of this powerful framework. Mastering advanced techniques further enhances performance, enabling the creation of high-performance applications across a wide range of domains. Further exploration of CUDA resources and best practices is encouraged to fully leverage its capabilities.

To learn more about parallel programming and GPU computing, consider exploring resources like the NVIDIA CUDA Zone and the Nvidia GPU Accelerated Computing website. You can also find numerous tutorials and courses on platforms like Coursera and Udemy.

--

--

Written by Mayin Dev

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

CUDA and GPU: The Dynamic Duo for Model Training

Ebad Sayed

Follow

--

Listen

Share

Using .to("cuda") to transfer data to the GPU is a common practice for accelerating mathematical operations. CPUs are quicker for single operations, but the strength of GPUs lies in their ability to perform parallel processing. CPUs are designed to execute a sequence of operations (threads) quicker but can execute only a few operations. On the other hand GPUs are built to handle millions of operations simultaneously, even though this comes at the cost of individual thread performance.

This capability allows us to perform operations in parallel. For instance, when adding elements of a million-sized tensor, instead of sequentially adding each element within a loop, a GPU can add all elements simultaneously. To achieve this, we use CUDA, a platform developed by NVIDIA that enables developers to incorporate GPU support into their software applications.

History of GPUs

Early Years: GPUs for Graphics

1980s-1990s: GPUs were initially developed to accelerate graphics rendering, primarily for video games and visual simulations. Companies like NVIDIA and Array Technologies Incorporated (ATI now part of Advanced Micro Devices AMD) led the way in developing powerful graphics cards.

Birth of CUDA: The Turning Point

2006: NVIDIA introduced CUDA (Compute Unified Device Architecture), a parallel computing platform and programming model. CUDA allowed developers to harness the parallel processing power of GPUs for general-purpose computing, marking the beginning of GPU acceleration beyond graphics.

GPUs Enter Deep Learning

2010s: Researchers began to explore GPUs for deep learning. The parallel architecture of GPUs proved to be highly efficient for the matrix and tensor computations essential in neural network training.2012: AlexNet, a convolutional neural network (CNN) that won the ImageNet Large Scale Visual Recognition Challenge, was trained using two NVIDIA GTX 580 GPUs. This success demonstrated the potential of GPUs in deep learning, sparking widespread interest.2014: Google introduced its TensorFlow framework, which included GPU support, making it easier for researchers and developers to leverage GPUs for deep learning tasks.

GPUs in the Era of Large Language Models (LLMs)

2018-Present: The development of large language models like BERT, GPT-2, and GPT-3 highlighted the necessity for massive computational power.2019: NVIDIA’s Turing architecture, with the release of the RTX 20 series and later the A100 based on the Ampere architecture, provided further enhancements in deep learning performance.2020: GPT-3, developed by OpenAI, demonstrated the scalability of deep learning models, requiring thousands of GPUs to train.

But how do deep learning algorithms take advantage of GPUs computation performance in practice? Let’s find out!

Deep learning models use mathematical operations like matrix multiplication, vector addition, etc. If we optimize these operations, we can improve the performance of these models.For example, addition of two arrays C = A + B. We can use CPU multithreading in order to run this computation in parallel. But it can only handle a few such threads simultaneously. Using GPUs we can run millions of threads simultaneously, which improves the performance of mathematical operations on huge vectors and matrices.

GPU vs CPU

Imagine you need to send several packages across a city. A motorcycle (CPU) can deliver a single package quickly, but if you have many packages to send, a delivery truck (GPU) can carry them all at once. Although the truck might be slower than the motorcycle for a single package, it becomes much more efficient when handling multiple packages at the same time.

CPUs are designed for fast sequential processing, making them ideal for tasks that require quick, single-threaded performance. While GPUs, on the other hand, excel in parallel processing, allowing them to handle many tasks simultaneously.

Comparison Video :- https://www.youtube.com/watch?v=-P28LKWTzrI

Understanding CPU and GPU Design Differences

CPU (Central Processing Unit) Design:

GPU (Graphical Processing Unit) Design:

Practical Implications

Single-Threaded Tasks (CPU)

Parallel Tasks (GPU)

Introduction to CUDA

When you execute a deep learning model, you’ll often opt for a commonly used Python library such as PyTorch or TensorFlow. These libraries are known to execute C/C++ code at their core. Furthermore, to enhance processing speed, you may harness the power of GPUs. This is where CUDA comes into the picture! CUDA, short for Compute Unified Device Architecture, is a platform designed by NVIDIA for general-purpose computing on their GPUs. It enables developers to leverage NVIDIA’s GPU computational capabilities in their software applications, going beyond just graphics rendering.To achieve this, CUDA offers a straightforward interface based on C/C++ (CUDA C/C++) that allows access to the GPU’s virtual instruction set and specific operations such as data transfer between the CPU and GPU.

Let’s compare CPU and GPU via coding

MultiplyVectorsOnCPU function is performing element-wise multiplication of two input arrays a and b, storing the result in array c. This operation is executed sequentially on the CPU.In CUDA, you would typically write a kernel to perform the same element-wise multiplication but execute it on the GPU. Each thread in a CUDA kernel is responsible for one element of the output array. The number of threads and how they are organized into blocks and grids is specified by the programmer and depends on the problem being solved and the hardware constraints.

To rewrite the MultiplyVectorsOnCPU function for execution on the GPU using CUDA, you would need to write a CUDA kernel and handle memory allocation and data transfer between the CPU and GPU. So let’s write the CUDA code for this operation:

Code for Single-Dimensional Threading

__global__ void MultiplyVectorsOnGPU(float *a, float *b, float *c, int N) {    int i = blockIdx.x * blockDim.x + threadIdx.x;    if (i < N) {        c[i] = a[i] * b[i];    }}int main() {    int N = 64000000; // 64 Million    size_t size = N * sizeof(float);    float *A, *B, *C;    cudaMallocManaged(&A, size);    cudaMallocManaged(&B, size);    cudaMallocManaged(&C, size);    // Initialize A and B    for (int i = 0; i < N; ++i) {        A[i] = 1.0f;        B[i] = 1.0f;    }    int blockSize = 256;    int numBlocks = (N + blockSize - 1) / blockSize;    cudaEvent_t start, stop;    cudaEventCreate(&start);    cudaEventCreate(&stop);    cudaEventRecord(start);    MultiplyVectorsOnGPU<<<numBlocks, blockSize>>>(A, B, C, N);    cudaEventRecord(stop);    cudaEventSynchronize(stop);    float milliseconds = 0;    cudaEventElapsedTime(&milliseconds, start, stop);    printf("Time taken for the execution of the task on GPU is %f milliseconds\n", milliseconds);    cudaFree(A);    cudaFree(B);    cudaFree(C);    return 0;}

MultiplyVectorsOnGPU() is a CUDA kernel function. The __global__ qualifier indicates that this function is called from the host (CPU) but executed on the device (GPU).

int i = blockIdx.x * blockDim.x + threadIdx.x;

Each thread in the CUDA kernel is responsible for one element of the output vector. This line of code calculates the index ‘i’ of the element to be processed by the current thread based on the block index ‘blockIdx.x’, the thread index within the block ‘threadIdx.x’, and the block size ‘blockDim.x’.Next we check if the calculated index i is within the bounds of the vectors N and perform element-wise multiplication of a[i] and b[i], storing the result in c[i].

size_t size = N * sizeof(float);

This calculates the total size in bytes needed to store the vectors (A, B, and C), considering each element as a float.

float *A, *B, *C;cudaMallocManaged(&A, size);cudaMallocManaged(&B, size);cudaMallocManaged(&C, size);

These lines allocate memory on the GPU for the input and output vectors using cudaMallocManaged, which allocates memory accessible from both the host and the device.

int blockSize = 256;int numBlocks = (N + blockSize - 1) / blockSize;

These lines determine the number of blocks needed to execute the CUDA kernel. The block size is set to 256 threads, and the number of blocks is calculated to ensure that all elements of the vectors are processed.In the provided code, we are launching ‘numBlocks’ blocks, each containing ‘blockSize’ threads. The total number of threads launched is numBlocks * blockSize. The numBlocks variable is calculated to ensure that all elements of the input vectors are processed. Each thread is responsible for one element of the output vector C.

To limit the maximum number of threads per block in CUDA, you can specify the block size when launching the kernel using the <<<…>>>syntax. The maximum number of threads per block is determined by the compute capability of your GPU.For example, to limit the block size to 256 threads per block:

int blockSize = 256;int numBlocks = (N + blockSize - 1) / blockSize;

In this code, blockSize is set to 256, and numBlocks is calculated to ensure that all elements of the vectors are processed. When you launch the kernel MultiplyVectorsOnGPU, each block will contain 256 threads, unless the total number of elements in your vectors is less than 256, in which case fewer threads will be launched.

cudaEvent_t start, stop;cudaEventCreate(&start);cudaEventCreate(&stop);

These lines create CUDA events start and stop for measuring the execution time.

cudaEventRecord(start);MultiplyVectorsOnGPU<<<numBlocks, blockSize>>>(A, B, C, N);cudaEventRecord(stop);cudaEventSynchronize(stop);

These lines record the start and stop events, call the CUDA kernel MultiplyVectorsOnGPU with the specified number of blocks and block size, and synchronize to ensure that all threads have finished execution before measuring the time.

float milliseconds = 0;cudaEventElapsedTime(&milliseconds, start, stop);

This calculates the elapsed time between the start and stop events in milliseconds.

cudaFree(A); cudaFree(B); cudaFree(C);

This frees the memory allocated for the vectors on the GPU.

Code for Multi-Dimensional Threading

__global__ void MultiplyVectorsOnGPU(float *a, float *b, float *c, int N) {    int idx = blockIdx.x * blockDim.x + threadIdx.x;    int idy = blockIdx.y * blockDim.y + threadIdx.y;    int offset = idy * N + idx;    if (idx < N && idy < N) {        c[offset] = a[offset] * b[offset];    }}int main() {    int N = 4096; // 4096x4096 matrix    size_t size = N * N * sizeof(float);    float *A, *B, *C;    cudaMallocManaged(&A, size);    cudaMallocManaged(&B, size);    cudaMallocManaged(&C, size);    // Initialize A and B    for (int i = 0; i < N * N; ++i) {        A[i] = 1.0f;        B[i] = 1.0f;    }    dim3 blockSize(32, 32);    dim3 numBlocks((N + blockSize.x - 1) / blockSize.x, (N + blockSize.y - 1) / blockSize.y);    cudaEvent_t start, stop;    cudaEventCreate(&start);    cudaEventCreate(&stop);    cudaEventRecord(start);    MultiplyVectorsOnGPU<<<numBlocks, blockSize>>>(A, B, C, N);    cudaEventRecord(stop);    cudaEventSynchronize(stop);    float milliseconds = 0;    cudaEventElapsedTime(&milliseconds, start, stop);    printf("Time taken for the execution of the task on GPU is %f milliseconds\n", milliseconds);    cudaFree(A);    cudaFree(B);    cudaFree(C);    return 0;}

The main difference between the code for multi-dimensional threading and single-dimensional threading lies in the use of multidimensional thread blocks and grids for processing a 2D matrix. This allows for a more natural and efficient processing of a 2D matrix on the GPU, compared to the previous code which processed 1D vectors.

Conclusion

In this article, we discussed fundamental concepts related to GPU acceleration and CUDA for improving the performance of deep learning models. This is just the beginning, and there is much more to explore. In this article we just got a clearer understanding of what happens behind the scenes when we use .to("cuda")to execute deep learning models on GPUs.

Further Readings

PyTorch from Scratch

CUDA Programming Guide

CUDANN Implementation

LLMs Training with CUDA in C++

--

--

Written by Ebad Sayed

I am currently a final year undergraduate at IIT Dhanbad, looking to help out aspiring AI/ML enthusiasts with easy AI/ML guides.

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

Generative AI

Home

Newsletter

About

Follow publication

All the latest news and updates on the rapidly evolving field of Generative AI space. From…

Follow publication

Accelerated Computing

CUDA Programming

Sam Mokhtari

Follow

Generative AI

--

Listen

Share

In previous modules, we covered the foundations of parallel computing and explored the capabilities of NVIDIA GPUs. This blog aims to build on that knowledge by introducing you to CUDA programming. We will explore the broader CUDA ecosystem, including key concepts, essential libraries, and techniques to improve the performance of your programs. By the end of this post, you will clearly understand how to leverage the power of CUDA for high-performance computing.

Subscribe and watch a detailed video on YouTube. Also, check out CUDA Programming Tutorials repository which is designed to help you learn NVIDIA CUDA programming through well-structured and practical examples.

Data Parallelism vs. Task Parallelism: The GPU-CPU Collaboration

Data parallelism involves applying the same operation to multiple data elements simultaneously. Imagine you have a large array of numbers, and you want to square each element. Each GPU thread could be responsible for squaring a single element, resulting in all the elements being processed simultaneously. This is what we mean by data parallelism — all threads perform the same task but on different pieces of data.

On the other hand, task parallelism is about executing different tasks independently — whether they’re working on the same dataset or separate ones. For instance, if you have several unrelated tasks, like image processing, network requests, and file I/O, you could assign each task to a separate CPU core to execute simultaneously. Task parallelism is more about dividing responsibilities than dividing data.

CPUs and multi-core systems are better suited for task parallelism because they can execute different instructions concurrently, giving them the flexibility to handle different types of tasks simultaneously.

GPUs are specifically designed for data parallelism. They use a model called SIMD (Single Instruction, Multiple Data), where all threads within a group, called a “warp,” execute the same instruction at the same time, but on different data. This approach aligns perfectly with the concept of data parallelism. In other words, GPUs are like an army of workers, each equipped to perform the same task on their individual piece of the workload, making them incredibly efficient for repetitive calculations.

CUDA leverages a heterogeneous programming model, meaning that the CPU (host) and GPU (device) work together to maximize efficiency. Each of them is assigned tasks that align with their strengths:

This division of labour ensures that the GPU can focus on the heavy lifting while the CPU manages and orchestrates the overall flow — kind of like a project manager (CPU) delegating tasks to an efficient team (GPU).

What is CUDA?

CUDA provides a comprehensive set of tools to take advantage of GPU power for high-performance computing. Here’s a look at some key components:

CUDA Toolkits

Writing CUDA Kernels: Where the Magic Happens

In CUDA, you write kernels, which are functions that run on the GPU. Kernels are like small, parallel tasks that can execute thousands of times concurrently across the GPU threads.

CUDA Execution Model

The CUDA execution model uses a hierarchical structure to organize and execute parallel tasks efficiently on the GPU. Let’s break it down:

Understanding Memory in CUDA Programming

Understanding and choosing the appropriate memory space is critical for achieving optimal performance in CUDA programming. Here’s a breakdown of the memory spaces and their characteristics:

Efficient memory usage directly impacts the performance of CUDA programs. By understanding the strengths and limitations of each memory type, you can maximize throughput, reduce latency, and fully leverage the GPU’s capabilities.

The CUDA Compiler (nvcc)

The CUDA compiler, nvcc, is a powerful tool that simplifies the process of preparing CUDA code for execution on a GPU. Let’s walk through its key features and functionality:

nvcc compiles the GPU code into PTX (Parallel Thread Execution), which is an intermediate representation, or directly into machine-specific binary code. It then links the host and device code into a single executable file that the program can run.

2. Common Command-Line Options: Developers can use various options with nvcc to customize the compilation process:

Offline Compilation: This method allows you to compile your code and link the pieces together later. It’s great for modular development where different parts of a program are developed independently.

Just-in-Time (JIT) Compilation: In some cases, PTX code is compiled into machine code at runtime. This provides flexibility, especially when targeting a variety of GPU architectures.

CUDA Runtime API vs. CUDA Driver API

CUDA offers two main APIs for interacting with the GPU: the CUDA Runtime API and the CUDA Driver API. Both serve important roles, but they cater to different levels of programming complexity and control.

The CUDA Runtime API is a high-level API that abstracts away many of the complexities involved in managing GPU resources. It is the most common choice for developers due to its simplicity and ease of use. The Runtime API handles many details like context management, memory allocation, and device setup automatically, which allows developers to focus more on writing efficient kernels rather than low-level GPU management.

Key Features of CUDA Runtime API:

The CUDA Runtime API is ideal for developers who want to get started quickly with GPU programming without diving into intricate details about hardware resources.

The CUDA Driver API provides a lower-level interface to the GPU, offering finer control over the hardware. Unlike the Runtime API, it requires explicit context management and provides more flexibility when managing GPU resources. The Driver API is often used in scenarios where maximum control and performance tuning are essential.

Key Features of CUDA Driver API:

The Driver API is ideal for developers who need more flexibility and control, such as when integrating CUDA into larger systems or when interacting closely with hardware resources.

CUDA Graphs

CUDA Graphs allow developers to define a sequence of operations — such as kernel launches, memory transfers, or other tasks — and treat them as a single, unified workload. Once a CUDA graph is created, it can be executed repeatedly without reissuing individual commands, which saves both time and reduces code complexity.

Benefits of Using CUDA Graphs

How to Create CUDA Graphs

There are two main ways to define a CUDA Graph:

By using CUDA Graphs, developers can create more optimized, less CPU-intensive workflows, maximizing the performance potential of both the CPU and GPU.

Dynamic Parallelism

Dynamic Parallelism is a powerful feature in CUDA that takes GPU programming to the next level. Let me explain how it works and why it’s so useful: Normally, the CPU (host) is responsible for launching kernels on the GPU (device). With Dynamic Parallelism, however, a kernel running on the GPU can launch and synchronize other kernels directly. This means the GPU can make decisions and create new tasks during execution without involving the CPU.

Why Is Dynamic Parallelism Useful?

•Supports Complex Program Structures: In applications with unpredictable data or workload requirements, Dynamic Parallelism allows kernels to create new tasks as needed. For example, in simulations, some regions of data might require more processing than others. Dynamic Parallelism lets the GPU adjust and handle these variations efficiently.

•Reduces CPU Overhead: Without Dynamic Parallelism, the CPU would need to manage and launch every kernel. This back-and-forth communication between the CPU and GPU can be slow. By letting the GPU launch kernels, we eliminate this bottleneck, improving performance.

•Adapts Dynamically: Kernels can respond to the current state of the computation and split the workload further, leading to better resource utilization and scalability.

Dynamic Parallelism unlocks new possibilities for designing efficient, scalable, and adaptive GPU programs, especially for applications like scientific simulations, hierarchical computations, and recursive algorithms.

Key CUDA Libraries for Accelerated Linear Algebra and Specialized Computations

CUDA libraries like cuBLAS, cuDNN, CUTLASS, cuFFT, and cuSPARSE are critical tools for high-performance computing on GPUs. Here’s an overview of their roles and benefits:

1. cuBLAS: Accelerating Linear Algebra

2. cuDNN: Driving Deep Learning

3. CUTLASS: Customizable Linear Algebra

4. cuFFT: Fast Fourier Transforms

5. cuSPARSE: Sparse Matrix Operations

Optimizing CUDA Programs for Maximum GPU Utilization

Profiling and debugging are critical steps in optimizing CUDA programs, allowing developers to pinpoint bottlenecks, improve performance, and ensure reliability. Here’s an in-depth look at the tools and techniques available:

1. Profiling CUDA Programs

Profiling helps identify inefficiencies in CUDA applications, enabling targeted optimizations for better GPU utilization and faster execution.

2. NVIDIA Nsight Systems

3. Command-line Profiler (nvprof)

A lightweight tool for quick profiling, capturing kernel execution times and memory transfer metrics.

2. Debugging CUDA Programs

Debugging in CUDA is complex due to the parallel nature of GPU programming. Tools like cuda-gdb simplify this process by providing powerful debugging features.

cuda-gdb is a CUDA-specific debugging tool that enables developers to:

Conclusion

CUDA offers a powerful framework for leveraging GPUs to tackle computationally intensive tasks. By understanding its core concepts, utilizing its rich ecosystem of libraries, and applying effective optimization techniques, developers can achieve exceptional performance for a wide range of applications.

This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.

Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!

--

--

Published in Generative AI

All the latest news and updates on the rapidly evolving field of Generative AI space. From cutting-edge research and developments in LLMs, text-to-image generators, to real-world applications, and the impact of generative AI on various industries.

Written by Sam Mokhtari

Technology thought leader with 15+ years in cloud, data analytics, and AI @ AWS | PhD | Author & Speaker | Life Mentor & Coach

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

CUDA programming: Custom MNIST MLP engine

Harsh Patel

Follow

--

Listen

Share

In this blog, I will guide you through how to code the cuda kernel for MNIST MLP inference.

Introduction:

In the ever-evolving landscape of artificial intelligence and deep learning, neural networks have emerged as a dominant force, propelling groundbreaking advancements across various domains. Among these neural network architectures, the Multilayer Perceptron (MLP) stands out as one of the most fundamental and widely-used models for its simplicity and effectiveness in solving various tasks. One such task is digit recognition, where the classic MNIST dataset has become the de facto benchmark for evaluating the performance of machine learning algorithms.

In this blog post, we embark on an exhilarating journey into the realm of high-performance computing and CUDA programming to unleash the full potential of custom MNIST MLP inference. NVIDIA’s CUDA (Compute Unified Device Architecture) has revolutionized the way we harness the power of Graphics Processing Units (GPUs) to accelerate computationally intensive tasks. By leveraging the parallel processing capabilities of GPUs, we can achieve significant speedups in training and inference times, making them indispensable tools for modern deep learning applications.

Our objective throughout this article is to demystify the complexities of CUDA programming and empower you to build a custom MLP inference engine tailored to the MNIST dataset. Whether you are a seasoned deep learning practitioner or a curious enthusiast eager to dive into the world of GPU acceleration, this guide will equip you with the essential knowledge to wield CUDA effectively.

The knowledge gained from this blog will not only empower you to optimize MNIST digit recognition but also serve as a stepping stone to explore more complex deep learning tasks that demand high computational power.

So, buckle up and get ready to harness the parallel processing capabilities of CUDA as we embark on an exhilarating journey to supercharge our custom MNIST MLP inference engine!

Training the Model and Saving Weights in NumPy:

In this section of the blog, we covered the initial steps of implementing custom MNIST MLP inference using CUDA. We began by setting up the MNIST dataset and then designing the architecture of our Multilayer Perceptron (MLP) model and training it using PyTorch.

To preserve the trained model’s learned parameters for future use and compatibility with other environments, we saved the optimized weights and biases in NumPy format.

Data reading and Model training:

import torchimport torch.nn as nnimport torch.nn.functional as Ffrom torchvision import datasets, transformsimport numpy as npimport matplotlib.pyplot as pltimport osif torch.cuda.is_available():    device = torch.device("cuda")else:    device = torch.device("cpu")print("Using PyTorch version:", torch.__version__, " Device:", device)batch_size = 32train_dataset = datasets.MNIST(    "./data", train=True, download=True, transform=transforms.ToTensor())validation_dataset = datasets.MNIST(    "./data", train=False, transform=transforms.ToTensor())train_loader = torch.utils.data.DataLoader(    dataset=train_dataset, batch_size=batch_size, shuffle=True)validation_loader = torch.utils.data.DataLoader(    dataset=validation_dataset, batch_size=batch_size, shuffle=False)for X_train, y_train in train_loader:    print("X_train:", X_train.size(), "type:", X_train.type())    print("y_train:", y_train.size(), "type:", y_train.type())    breakclass Net(nn.Module):    def __init__(self):        super(Net, self).__init__()        self.fc1 = nn.Linear(28 * 28, 50)        self.fc1_drop = nn.Dropout(0.2)        self.fc2 = nn.Linear(50, 50)        self.fc2_drop = nn.Dropout(0.2)        self.fc3 = nn.Linear(50, 10)    def forward(self, x):        x = x.view(-1, 28 * 28)        x = F.relu(self.fc1(x))        x = self.fc1_drop(x)        x = F.relu(self.fc2(x))        x = self.fc2_drop(x)        return F.log_softmax(self.fc3(x), dim=1)model = Net().to(device)optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.5)criterion = nn.CrossEntropyLoss()print(model)def train(epoch, log_interval=200):    # Set model to training mode    model.train()    # Loop over each batch from the training set    for batch_idx, (data, target) in enumerate(train_loader):        data = data.to(device)        target = target.to(device)        optimizer.zero_grad()        output = model(data)        loss = criterion(output, target)        loss.backward()        optimizer.step()        if batch_idx % log_interval == 0:            print(                "Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}".format(                    epoch,                    batch_idx * len(data),                    len(train_loader.dataset),                    100.0 * batch_idx / len(train_loader),                    loss.data.item(),                )            )def validate(loss_vector, accuracy_vector):    model.eval()    val_loss, correct = 0, 0    for data, target in validation_loader:        data = data.to(device)        target = target.to(device)        output = model(data)        val_loss += criterion(output, target).data.item()        pred = output.data.max(1)[1]  # get the index of the max log-probability        correct += pred.eq(target.data).cpu().sum()    val_loss /= len(validation_loader)    loss_vector.append(val_loss)    accuracy = 100.0 * correct.to(torch.float32) / len(validation_loader.dataset)    accuracy_vector.append(accuracy)    print(        "\nValidation set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n".format(            val_loss, correct, len(validation_loader.dataset), accuracy        )    )epochs = 5lossv, accv = [], []for epoch in range(1, epochs + 1):    train(epoch)    validate(lossv, accv)

Weights preserving:

weights = [    (name, param.detach().cpu().numpy()) for name, param in model.named_parameters()]for w in weights:    with open("./model/" + str(w[0]) + ".npy", "wb") as f:        print("shape")        if "bias" in w[0]:            weights = np.expand_dims(w[1], axis=(0))            print(weights.shape)            weights = np.ascontiguousarray(weights)            np.save(f, weights)        else:            weights = w[1].T            print(weights.shape)            weights = np.ascontiguousarray(weights)            np.save(f, weights)

Loading NumPy Weights in CUDA in C++:

In this section we will read the numpy weight in C++ using python interpeter. As we ready using Python interpeter we get PyArrayObject , which we will convert it into float. And we will move the float array to GPU using CUDA.

#include <stdio.h>#include <stdlib.h>#include <Python.h>#include <cuda_runtime.h>#include <numpy/arrayobject.h>#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSIONstruct Weights {    float* matrix;    int ndims;    int *shape;    long int size;};PyObject* read_numpy_file(const char* file_path) {    // Import the Python module containing the function    PyObject* numpy_module = PyImport_ImportModule("numpy");    if (numpy_module == nullptr) {        PyErr_Print();        return nullptr;    }    // Get the reference to the function    PyObject* numpy_function = PyObject_GetAttrString(numpy_module, "load");    if (numpy_function == nullptr) {        PyErr_Print();        Py_DECREF(numpy_module);        return nullptr;    }    // Create the arguments tuple    PyObject* args = PyTuple_New(1);    PyTuple_SetItem(args, 0, PyUnicode_FromString(file_path));    // Call the Python function with the arguments    PyObject* result = PyObject_CallObject(numpy_function, args);    if (result == nullptr) {        PyErr_Print();        Py_DECREF(numpy_module);        Py_DECREF(numpy_function);        Py_DECREF(args);        return nullptr;    }    // Print the shape of the NumPy array    PyObject* shape = PyObject_GetAttrString(result, "shape");    if (shape != nullptr) {        PyObject* repr = PyObject_Repr(shape);        const char* str = PyUnicode_AsUTF8(repr);        printf("Shape: %s\n", str);        Py_DECREF(repr);        Py_DECREF(shape);    } else {        printf("Failed to get shape.\n");    }    // Clean up references    Py_DECREF(numpy_module);    Py_DECREF(numpy_function);    Py_DECREF(args);    return result;}PyArrayObject* read_weights_from_numpy(const char* file_path, int print) {    // Call the Python function and get the PyObject reference to the NumPy array    PyObject* numpy_array = read_numpy_file(file_path);    if (numpy_array == nullptr) {        // Handle error        return nullptr;    }    // Use the PyObject reference to the NumPy array as needed    // Example: Print the array    if (print == 1){        PyObject* repr = PyObject_Repr(numpy_array);        const char* str = PyUnicode_AsUTF8(repr);        printf("%s\n", str);    }    // Convert the result to a NumPy array    PyArrayObject* array = reinterpret_cast<PyArrayObject*>(numpy_array);    return array;}int get_numpy_ndims(PyArrayObject* array){    // Get number of dimensions    int ndim = PyArray_NDIM(array);    printf("%d \n", ndim);    return ndim;}long int get_numpy_size(PyArrayObject* array){    // Get the total size of the array    npy_intp total_size_intp = PyArray_SIZE(array);    printf("%" NPY_INTP_FMT "\n", total_size_intp);        long int total_size = static_cast<long int>(total_size_intp);    return total_size;}float* convert_PyArrayObject_to_float(PyArrayObject* array, int print, int *shape, int ndim) {    printf("values %d %d \n", PyArray_TYPE(array), NPY_DOUBLE);    // Check the data type of the numpy array    if (PyArray_TYPE(array) != NPY_FLOAT32) {        printf("Input numpy array is not of type float.\n");    }    // Convert the weight into float    float* matrix = static_cast<float*>(PyArray_DATA(array));    // Printing weights for checking it after conversion     if (print == 1){        if (ndim == 2){            for (int i = 0; i < shape[0]; ++i) {                if (i ==0){                    for (int j = 0; j < shape[1]; ++j) {                        // matrix[j+(i*shape[1])] = 0.5f;                        if(j<3 || j>shape[1]-3){                            printf("%d index %.5f ", j+(i*shape[1]), matrix[j+(i*shape[1])]);                        }                    }                    printf("\n");                }            }        }        else if (ndim == 1){            for (int i = 0; i < shape[0]; ++i) {                // matrix[i] = 0.5f;                if(i<5 || i>shape[0]-5){                    printf("%d index %.5f ", i, matrix[i]);                }                            }            printf("\n");        }           }    return matrix; }float* move_weight_to_cuda(float* weights ,long int total_size){    // Allocate CUDA device memory    float* d_data;    printf("total size %d \n", total_size);    cudaError_t cudaStatus;    cudaStatus = cudaMalloc((void**)&d_data, total_size * sizeof(float));    if (cudaStatus != cudaSuccess) {        fprintf(stderr, "cudaMalloc failed: %s\n", cudaGetErrorString(cudaStatus));        // Handle the error or return an error code    }    // Copy the array data from host (CPU) to device (CUDA)    cudaStatus = cudaMemcpy(d_data, weights, total_size * sizeof(float), cudaMemcpyHostToDevice);    if (cudaStatus != cudaSuccess) {        fprintf(stderr, "cudaMalloc failed: %s\n", cudaGetErrorString(cudaStatus));        // Handle the error or return an error code    }    return d_data;}Weights read_weights(const char* file_path, int print){    Weights weight;    // reading numpy weights    PyArrayObject* array = read_weights_from_numpy(file_path, print);    if (array == nullptr) {        // Handle error        weight.matrix = nullptr;        return weight;    }    int ndims = get_numpy_ndims(array);    get_numpy_shape(array, weight, ndims);        long int size = get_numpy_size(array);    float* matrix = convert_PyArrayObject_to_float(array, print, weight.shape, ndims);        // Release the PyObject reference    float* cuda_weights = move_weight_to_cuda(matrix, size);    Py_DECREF(array);    weight.ndims = ndims;    weight.size = size;    weight.matrix = cuda_weights;    return weight;}

Defining CUDA kernels for MLP in C++:

We have to define 3 kernel for make MLP to work matrixMulKernel, matrixAddKernel and softmaxKernel. For more details please check links in references.

__global__ void matrixMulKernel(float* matrixA, float* matrixB, float* matrixC, int rowsA, int colsA, int colsB) {    int row = blockIdx.y * blockDim.y + threadIdx.y;    int col = blockIdx.x * blockDim.x + threadIdx.x;    if (row < rowsA && col < colsB) {        float sum = 0.0f;        for (int k = 0; k < colsA; k++) {            sum += matrixA[row * colsA + k] * matrixB[k * colsB + col];        }        matrixC[row * colsB + col] = sum;        // Print intermediate result        // printf("Intermediate result at rowsA %d, colsB %d  [%d][%d]: %.5f\n", rowsA, colsB, row, col, sum);    }}__global__ void matrixAddKernel(float* matrixA, float* matrixB, float* matrixC, int rows, int cols) {    int row = blockIdx.y * blockDim.y + threadIdx.y;    int col = blockIdx.x * blockDim.x + threadIdx.x;    if (row < rows && col < cols) {        int index = row * cols + col;        matrixC[index] = matrixA[index] + matrixB[index];    }}__global__ void softmaxKernel(float* input, float* output, int rows, int cols) {    int row = blockIdx.y * blockDim.y + threadIdx.y;    int col = blockIdx.x * blockDim.x + threadIdx.x;    if (row < rows && col < cols) {        int index = row * cols + col;        // Compute the exponential of each element        float expVal = expf(input[index]);        // Compute the sum of exponentials for the row        float sumExp = 0.0f;        for (int i = 0; i < cols; ++i) {            sumExp += expf(input[row * cols + i]);        }        // Compute the softmax value for the element        output[index] = expVal / sumExp;    }}float* matrixMul(float* matrixA, float* matrixB, int rowsA, int colsA, int rowsB, int colsB){    float* matrixC;    printf(" ROW A %d  COLS A %d \n",rowsA, colsA);    printf(" ROW B %d  COLS B %d \n",rowsB, colsB);    cudaMalloc((void **)&matrixC, rowsA * colsB * sizeof(float));    dim3 blockSize(16, 16);    dim3 gridSize((colsB + blockSize.x - 1) / blockSize.x, (rowsA + blockSize.y - 1) / blockSize.y);    matrixMulKernel<<<gridSize, blockSize>>>(matrixA, matrixB, matrixC, rowsA, colsA, colsB);    return matrixC;}float* matrixAdd(float* matrixA, float* matrixB, int rows, int cols){    float* matrixC;    printf(" ROW %d  COLS %d \n",rows, cols);    cudaMalloc((void **)&matrixC, rows * cols * sizeof(float));    dim3 blockSize(16, 16);    dim3 gridSize((cols + blockSize.x - 1) / blockSize.x, (rows + blockSize.y - 1) / blockSize.y);    matrixAddKernel<<<gridSize, blockSize>>>(matrixA, matrixB, matrixC, rows, cols);    return matrixC;}float* softmax(float* input, int rows, int cols){    float* matrixC;    printf(" ROW %d  COLS %d \n",rows, cols);    cudaMalloc((void **)&matrixC, rows * cols * sizeof(float));    dim3 blockSize(16, 16);    dim3 gridSize((cols + blockSize.x - 1) / blockSize.x, (rows + blockSize.y - 1) / blockSize.y);    softmaxKernel<<<gridSize, blockSize>>>(input, matrixC, rows, cols);    return matrixC;}

Defining main function for inference in C++:

In this section, we will combine all the reading and kernel defination for MLP inference on CUDA.

int main() {    // Initialize the Python interpreter    Py_Initialize();    // Ensure that NumPy is available    import_array();    Weights fc1_w = read_weights("./mlp/mnist_mlp/model/fc1.weight.npy", 0);    Weights fc2_w = read_weights("./mlp/mnist_mlp/model/fc2.weight.npy", 0);    Weights fc3_w = read_weights("./mlp/mnist_mlp/model/fc3.weight.npy", 0);    printf("#############################\n");    Weights fc1_b = read_weights("./mlp/mnist_mlp/model/fc1.bias.npy", 0);    Weights fc2_b = read_weights("./mlp/mnist_mlp/model/fc2.bias.npy", 0);    Weights fc3_b = read_weights("./mlp/mnist_mlp/model/fc3.bias.npy", 0);    //Read image    Inputs image = read_image("./mlp/mnist_mlp/images/1.npy", 0);    // Inputs image = read_image("./mlp/mnist_mlp/2.npy", 0);    // Finalize the Python interpreter    Py_Finalize();    int output_row = 1;    int output_col = 10;    float* matrixC = nullptr;        matrixC = matrixMul(image.matrix, fc1_w.matrix, image.shape[0], image.shape[1], fc1_w.shape[0], fc1_w.shape[1]);    matrixC = matrixAdd(matrixC, fc1_b.matrix, fc1_b.shape[0], fc1_b.shape[1]);    matrixC = matrixMul(matrixC, fc2_w.matrix, fc1_b.shape[0], fc1_b.shape[1], fc2_w.shape[0], fc2_w.shape[1]);    matrixC = matrixAdd(matrixC, fc2_b.matrix, fc2_b.shape[0], fc2_b.shape[1]);    matrixC = matrixMul(matrixC, fc3_w.matrix, fc2_b.shape[0], fc2_b.shape[1], fc3_w.shape[0], fc3_w.shape[1]);    matrixC = matrixAdd(matrixC, fc3_b.matrix, fc3_b.shape[0], fc3_b.shape[1]);    matrixC = softmax(matrixC, fc3_b.shape[0], fc3_b.shape[1]);    printf("output shape 1: %d, %d", fc3_b.shape[0], fc3_b.shape[1]);    float* C = (float *)malloc(output_row * output_col * sizeof(float));    cudaMemcpy(C, matrixC,  output_row * output_col * sizeof(float), cudaMemcpyDeviceToHost);    cudaDeviceSynchronize();    printf("rowsA %d\n", output_row);    printf("colsB %d\n", output_col);    for (int i = 0; i < output_row; i++) {        for (int j = 0; j < output_col; j++){            printf("%f ", C[i * output_row + j]);                    }        printf("\n");    }    sleep(5);     // Clean up CUDA device memory    cudaFree(fc1_w.matrix);    cudaFree(fc2_w.matrix);    cudaFree(fc3_w.matrix);        cudaFree(fc1_b.matrix);    cudaFree(fc2_b.matrix);    cudaFree(fc3_b.matrix);    return 0;}

Compile and Running:

To compile the program, we need to use the “nvcc” compiler provided by the CUDA Toolkit. We can compile the program with the following command:

nvcc -o ml.o ml.cu -I/usr/include/python3.8/ -I/user/lib/python3/dist-packages/numpy/ -lpython3.8

To run the program, we simply execute the binary file generated by the compiler:

./ml.o

Conclusion:

I hope this blog has given you a good introduction to CUDA programming with C, and that you’re excited to explore more advanced topics in CUDA programming. Happy coding!

References:

--

--

Written by Harsh Patel

Computer Science || Android, Computer Vision || Blockchain

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

Power of GPU’s for General-Purpose using NVIDIA’s CUDA Architecture

Vigneshwara

Follow

--

Listen

Share

CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model developed by NVIDIA. It allows developers to harness the immense power of NVIDIA GPUs (graphics processing units) for tasks beyond their traditional role in graphics rendering.

Why CUDA? The Need for Parallelism

Traditional CPUs (central processing units) excel at sequential tasks, processing instructions one after another. However, many real-world problems, such as:

involve massive amounts of data and parallel operations. This is where GPUs shine! They have thousands of cores designed to handle many computations concurrently. CUDA bridges the gap, enabling programmers to use languages like C, C++, and Fortran to write code that runs on GPUs.

Key Concepts in CUDA

GPU as a Co-Processor:

CUDA views the GPU as a co-processor working alongside the CPU. The CPU manages the overall program flow, while the GPU handles the heavy parallel computations.

Kernels:

A kernel is a function written in CUDA C/C++ that executes on the GPU. It’s designed to be highly parallel, with each thread of the kernel performing the same operation on different parts of the data.

Example Kernel Code (Vector Addition): This is a classic example that demonstrates parallel computation. Each thread in the kernel adds corresponding elements from two input arrays and stores the result in an output array.

#include <cuda.h>#include <stdio.h>// Kernel definition for vector addition__global__ void vectorAdd(float *a, float *b, float *c, int n) {  int i = blockIdx.x * blockDim.x + threadIdx.x;  if (i < n) {    c[i] = a[i] + b[i];  }}int main() {  // Allocate memory on the host (CPU)  int n = 1024;  float *a, *b, *c;  a = (float *)malloc(n * sizeof(float));  b = (float *)malloc(n * sizeof(float));  c = (float *)malloc(n * sizeof(float));  // Initialize vectors on the host  for (int i = 0; i < n; ++i) {    a[i] = i;    b[i] = 2*i;  }  // Allocate memory on the device (GPU)  float *d_a, *d_b, *d_c;  cudaMalloc((void **)&d_a, n * sizeof(float));  cudaMalloc((void **)&d_b, n * sizeof(float));  cudaMalloc((void **)&d_c, n * sizeof(float));  // Copy data from host to device  cudaMemcpy(d_a, a, n * sizeof(float), cudaMemcpyHostToDevice);  cudaMemcpy(d_b, b, n * sizeof(float), cudaMemcpyHostToDevice);  // Define grid and block dimensions  int threadsPerBlock = 256;  int blocksPerGrid = (n + threadsPerBlock - 1) / threadsPerBlock;  // Launch the kernel  vectorAdd<<<blocksPerGrid, threadsPerBlock>>>(d_a, d_b, d_c, n);  // Copy result from device to host  cudaMemcpy(c, d_c, n * sizeof(float), cudaMemcpyDeviceToHost);  // Print the result (for verification)  for (int i = 0; i < 10; ++i) {    printf("c[%d] = %f\n", i, c[i]);   }  // Free memory on the device  cudaFree(d_a);  cudaFree(d_b);  cudaFree(d_c);  // Free memory on the host  free(a);  free(b);  free(c);  return 0;}

Threads, Blocks, and Grids:

CUDA organizes parallel execution using a hierarchy:- Thread: The smallest unit of execution on the GPU.- Block: A group of threads that execute together.- Grid: A collection of blocks that defines the overall parallel execution structure.

Memory Hierarchy:

CUDA provides different types of memory:- Global Memory: Large, but relatively slow, accessible by all threads.- Shared Memory: Smaller, faster memory shared by threads within a block, used for efficient data sharing.- Constant Memory: Read-only memory for data that doesn’t change, cached for fast access.- Texture Memory: Optimized for spatial locality, used for tasks like image processing.

Data Transfer:

Moving data between the CPU’s main memory and the GPU’s memory is a key aspect of CUDA programming. It’s important to minimize data transfers to optimize performance.

CUDA Programming Model

Advantages of CUDA

Challenges of CUDA

Beyond the Basics — Advanced CUDA Concepts

Conclusion: The Power and Potential of CUDA

CUDA empowers developers to take advantage of the immense parallel processing capabilities of GPUs, revolutionizing fields like scientific computing, AI, and high-performance data processing. While it requires learning new concepts and tools, the potential for dramatic performance gains makes CUDA a compelling choice for modern computing.

To learn more about CUDA:

Note: Let me know for any in-depth explanations.

CUDA empowers developers to take advantage of the immense parallel processing capabilities of GPUs, revolutionizing fields like scientific computing, AI, and high-performance data processing. While it requires learning new concepts and tools, the potential for dramatic performance gains makes CUDA a compelling choice for modern computing.

--

--

Written by Vigneshwara

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

GPGPU

Home

About

The best of GPGPU development articles, tutorials, and news

Block Sparse Matrix-Vector Multiplication with CUDA

Georgii Evtushenko

Follow

GPGPU

--

Listen

Share

In the previous post, we’ve discussed sparse matrix-vector multiplication. It was shown that it’s possible to take advantage of knowledge about a position of zeroes by storing matrices in special data structures. Although we’ve improved performance and memory space requirements, we haven’t used all the information about zeroes position. In some applications, non-zeroes are gathered in blocks. The knowledge about these blocks could give us more room for optimization. In this post, I’m going to discuss the efficiency of block sparse matrix-vector multiplication on GPU. To show some real-live application results, I develop a Matrix Structural Analysis application, which is used to simulate the Golden Gate bridge structure.

Block Compressed Sparse Row (BCSR)

BCSR is one of the most popular block sparse matrix formats. In BCSR, all blocks have the same size. To understand this format imagine a sparse matrix with the block size equal to one. In this case, CSR and BCSR matrix representations are equivalent. Block size increasing doesn’t affect the column and row pointer arrays. Instead, it just extends the values array. That is, columns and row pointer arrays contain values for blocks. Blocks are stored in the value array contiguously. The modification of CSR dramatically reduces memory space requirements.

To access block rows data, we need to get a number of blocks before the block row (row_ptr[block_row]) and multiply this value by block size square. It’s possible to store blocks in row-major or column-major order. As I’ll show further, elements order within blocks might affect performance. I’ll start with row-major order.

In listing 1 I assign one thread per one non-block row. To test the performance of BCSR implementations I’ve generated N-diagonal block sparse matrices. These matrices are suited for now, because we want to exclude load balancing issues.

The best speedup of this BCSR SpMV is about 65.8 (for block size equal to 16). Although the kernel achieves coalesced access to data of matrix for small block sizes, it drops performance for block size equal to 32. I used float matrices in this post, so each thread in the warp read data in an exact cache line of 128 bytes.

In contrast, cuSPARSE implementation of SpMV for block sparse matrices doesn’t seem to have such a dramatic performance drop. It means that we have some room for optimizations. The obvious solution is to assign warp per non-block row. That would give us fully-coalesced memory access.

This optimization improves the performance in case of matrices with big block sizes. Although it is cheap to perform inner warp reduction, the new version drops performance for small blocks.

The less obvious optimization is to use column-major order within each block.

This change allows coalesced access to block. That means that threads access data of block consecutively.

Now it might be noticed, that threads read vector x consecutively. We could try to cache it.

But it doesn’t affect performance at all. Is it the limit? I don’t think so.

We could allocate warp per block row in another way. The new algorithm is claimed to be more efficient in handling matrices with block sizes that are not powers of two. In the algorithm warp iterates through its block row’s columns, covering 32/bs columns at a time. The algorithm limits max block size by warp size. Although this optimization would limit matrix block size, it might be quite interesting to try.

The algorithm must compute a block index and a horizontal position within the block. In the paper, where I found this algorithm, it’s noted that the algorithm comes at the cost of increased latency from integer operations. Is it possible that the column-by-column algorithm performance (fig. 6) is limited by integer division operations?

The NVIDIA best practices guide states “Integer division and modulo operations are particularly costly and should be avoided or replaced with bitwise operations whenever possible”. To improve performance I’ve tried int_fastdiv library. Unfortunately, it doesn’t improve performance a lot (fig. 7).

Is it possible that shift operations solve our performance issue?

Although it does outperform int_fastdiv (fig. 8), it’s still not enough.

Is there a way for further optimization of integer divisions? Well, it’s better to not perform them at all. Let’s perform division at compile-time.

The templated version is quite close to our best version. Its performance is achieved not only by integer division optimization. The compiler also unrolled inner loops.

I think it’s time to stop using ideal cases of N-diagonal matrices. To test the proposed kernels on real-life matrix structures I’ve selected matrix structural analysis area.

BCSR with non-uniform pattern and matrix structural analysis

Structural analysis is the process of predicting the displacements within a given structure under a prescribed loading condition. I’m not going to describe a matrix structural analysis here. Instead, I’m going to describe the way a mesh affects a matrix structure. To get the idea of global stiffness matrix formation we need to understand the structure of a member stiffness matrix k for each element in mesh:

where L denotes length, E — Young’s modulus of elasticity and A — cross-sectional area of an element. Also, we need a transformation matrix T.

To get a member stiffness matrix for the particular element in global coordinates we need to perform:

That gives us a 4x4 matrix. To assemble it into a global stiffness matrix we need to consider K elements position. The first two rows of the K matrix are assigned for the first element’s node. The first two columns of the first two rows are to be added into the first nodes’ diagonal block. The second two columns of the first two rows are to be added into the first node’s off-diagonal block. The same logic is applied for the second two rows and the second node of the element. That gives us the global stiffness block matrix, which blocks size are equal to 2 and which rows and columns count is equal to nodes count of a mesh. It’s possible to increase the block size of the stiffness matrix by considering a plane frame structure instead of plane trusses. To compute plane frame structure, k and T matrices are to be changed:

where I denotes a moment of inertia. To vary a stiffness matrix size I’ve written parametrized Golden Gate Bridge structure generator.

There is a repeating structure in this mesh. So it’s easy to vary segments count or length of the bridge to control global stiffness matrix size.

The Golden Gate Bridge produces a matrix structure that is shown in figure 11. It has a more irregular pattern than a simple N-diagonal matrix.

The performance of different BSpMV kernels is presented in figure 12. In my experiments, the column-major sparse matrix outperformed other options when I assigned thread per non-block row.

It’s important to note that load disbalance questions weren’t considered in the scope of this post. I hope to consider this questions in one of the following posts. As always, you could find the source codes for this post on github.

--

--

Published in GPGPU

The best of GPGPU development articles, tutorials, and news

Written by Georgii Evtushenko

C++ Software Developer with experience in high-performance computing.

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams


Sign up

Sign in

Sign up

Sign in

Bellman-Ford Single Source Shortest Path Algorithm on GPU using CUDA

Parallel algorithm to compute the shortest distance from a source vertex to all other vertices in a connected graph

Raj Sengo

Follow

TDS Archive

--

1

Listen

Share

Overview

Traversing large graphs to compute different information has various use cases in the real-world like social media networks, communication networks, search engine indexing & page rank, VLSI design and biological network analysis. Bellman-Ford, Dijkstra’s and Delta Stepping are widely used Single Source Shortest Path Algorithm (SSSP) algorithms. Dijkstra’s algorithm provides a work efficient implementation, whereas Bellman-Ford provides scope for easy parallel implementation. Delta Stepping algorithm introduces a trade-off between the two. This article presents three parallel implementation techniques to accelerate the Bellman-Ford SSSP algorithm using Compute Unified Device Architecture (CUDA) on General-purpose computing on graphics processing units (GPGPU). We also compare the performance of all three variations on large graphs. We observed that the run time of a sequential implementation can be reduced by 99.4% for a sparse graph with 400k vertices and 1 million edges with optimized CUDA implementation.

What is Single Source Shortest Path (SSSP)?

A Graph G = (V, E) where V is set of vertices and E is set of Edges. An edge (u,v,w) is a path from node u to v and its weight is w. If we think the nodes as cities, then the edges are routes between the cities and weights are the distance between the cities.

In the example graph shown below V = {A, B, C, D, E} and E = {(A,B,9), (A,C,4), (B,C,10), (B,D,2), (B,E,3), (C,D,2), (C,E,11), (D,B,2),(E,D,2)}. Single source shortest path is a shortest distance from a source vertex e.g. “A” to all other vertices.

In the above example graph, SSSP from source vertex “A” to all other vertices is given by the blue arrows. i.e A → C → D → B → E

Data set

We used two different types of large graphs. USA road network graph data sets from DIMACS Shortest Paths Implementation challenge (The Center for Discrete Mathematics and Theoretical Computer Science. 9th DIMACS Implementation Challenge) and random generated graphs using SPRAND tool by Cherkassky, Goldberg and Radzik (Reference [1])

Graph Representation

A graph G(V, E) is generally represented via an adjacency matrix or adjacency list. For a sparse graph such as road networks, adjacency list is the preferred representation, since it takes less space. Compressed Sparse Row (CSR) representation is an alternate form of an adjacency list, in which the lists of vertices are packed into one single large array. We found that this representation is suitable for CUDA implementation (Reference [2], [3]). As shown in the below figure, four arrays are used to represent the graph; a vertex array V that stores all the vertices, an index array I that stores the starting position of the adjacency list of edges for each V[i], an edge array E, and a weight array W that stores the weights of each edge. I[i+1]−I[i] provides the number of edges of V[i]

GPU and CUDA

An increase in the use of general-purpose computing on graphics processing units (GPGPU) gives us massive scale parallel computing capabilities. The GPGPU is based on single instruction, multiple thread (SIMT) execution model in which each thread executes the same code. Compute Unified Device Architecture CUDA (Compute Unified Device Architecture) is a parallel computing platform and APIs created by Nvidia. It exposes GPU parallelism for general-purpose computing and retains performance. It’s developed based on industry-standard C++. CUDA consists of a small set of extensions to enable heterogeneous programming.

The GPGPU consists of several streaming multiprocessors (SMs). Each SM consists of several streaming processors(SPs). Each SM has its memory, known as shared memory, which is shared across all the SPs in the SM. From a developer’s standpoint GPUs can be viewed as grids, SMs can be viewed as blocks and SPs can be viewed as threads. The kernel is the piece of code executed by threads. Each thread has its ID that plays a vital role in determining which part of the input data to be accessed by the thread.

Bellman-Ford Sequential Algorithm

Bellman-Ford algorithm is a simple algorithm which uses edge relaxation technique. It relaxes all the edges, | V | − 1 times. Where | V | is the number of vertices in the graph. It can also work on the graph with negative weight edges provided there is no negative edge cycle. For this study, only graphs with positive edge weights were considered.

What is edge relaxation?

It’s a technique to correct the approximate distances with the correct ones. In the following graph below, d[u] is the distance from source “s” to “u” and d[v] is the distance from source “s” to “v”. There is an edge between “u” and “v” with weight “w”. if the distance of v is more than the distance of u + weight of (u, v) then the distance of v is updated with the distance of u + weight of (u, v).

Bellman-Ford Parallel CUDA Implementations

Version 1 — Monolithic kernel

In our first approach, we introduced a monolithic CUDA kernel in which each vertex of the graph is assigned to a separate thread. Each thread relaxes all the outgoing edges of the vertex identified by the thread ID. In this approach, the number of GPU blocks is calculated during run time based on the number of vertices in the input graph. This type of kernel assumes we have adequate threads to cover all vertices of the input graph.

Version 2 — Kernel with Grid-Stride loop

In our second technique to accelerate the parallel implementation, we used grid stride in the CUDA kernel. The stride is calculated using blockDim.x * gridDim.x, which is equal to the total number threads in the grid. As an example, if there are 1024 threads in the grid, thread 0 will process the vertex at indices 0, 1024, 2048, etc. This grid stride loop approach provides scalability and thread reuse. It also ensures that no thread is idle and every thread does an equal amount of work

Version 3 — Kernel with Grid-Stride loop and relax edges only when needed

To further optimize the performance, we introduced a boolean array F of size |V|. This array is initialized to false at the beginning for all vertices of the graph. The source is set to true. When a vertex is updated with a shorter distance from the source as part of the relaxation, its corresponding flag in the F array is set to true. This indicates that all the outgoing edges from that vertex need further relaxation in the next iteration. The relax kernel uses this information and relaxes only if the flag is set to true for the corresponding vertex. This approach combined with the grid-stride loop reduces the work done by each thread that is not necessary hence ensures further speed up of the overall execution.

Test Environment

We used the Texas Advanced Computing Center (TACC) Maverick2 supercomputer system to analyze the performance of all our implementations on large graphs. Maverick2 has Nividia GeForce GTX 1080 Ti GPU device with 28 SMs and 128 SPs per SM and 11GB of global memory. Each SM has around 49 KB of shared memory.

Performance Analysis

Our analysis reveals that version 3 of the implementation performed the best. The performance was even better on randomly generated dense graphs in comparison with sparse real-world road network graphs. First, we recorded that the basic sequential implementation took 3.25 hours to perform SSSP calculation on a sparse New York road network graph with 400k+ vertices, and over 1 million edges. In comparison, CUDA version-1 took 39 seconds, version-2 took 42 seconds, and version-3 took 16 seconds to compute the same. We also observed that version 2 i.e grid-stride loop kernel performed better for larger graphs with more than 1 million vertices and 4 million edges. The performance was even better for the randomly generated dense graphs. This indicates that the overhead of launching a new thread block reduces with the introduction of the grid-stride loop, which ensures the block size does not increase as the input increases, thus allowing each thread to handle more work. The table below shows the elapsed run time of our Bellman-Ford implementations for various large graphs.

Conclusion

We presented three variations of parallel implementations of Bellman-Ford single source shortest path algorithm on GPU using CUDA. The kernel with grid-stride loop and the logic to relax edges only when needed, performed better for larger graphs from 1–10 million edges. We used CSR representation in our implementations. This approach takes more space for dense graphs. Further study can be done to use an alternate format (like adjacency matrix) for dense graphs and use of shared memory inside the CUDA blocks to study the performance.

This blog post is based on a term paper that I and my partner Stephan Garland submitted for the Parallel Algorithms class at The University of Texas, Austin, Summer 2020. Many thanks to Dr. Vijay Garg for the well structured and wonderfully taught Parallel Algorithms class.

Github link: https://github.com/sengorajkumar/gpu_graph_algorithms

References

[1]Boris Cherkassky, Andrew V. Goldberg and Tomasz Radzik. Shortest Paths Algorithms: Theory And Experimental Evaluation. Mathematical Programming, pages 129–174. 1993.

[2]Pankhari Agarwal and Maitreyee Dutta. New Approach of Bellman-Ford Algorithm on GPU using Compute Unified Design Architecture (CUDA). International Journal of Computer Applications (0975–8887, 110 — №13.2015.

[3]Pawan Harish and P. J. Narayanan. Accelerating large graph algorithms on the GPU using CUDA. Center for Visual Information Technology, International Institute of Information Technology Hyderabad, INDIA.

[4]Mark Harris. CUDA Pro Tip: Write Flexible Kernels with Grid-Stride Loops.https://developer.nvidia.com/blog/cuda-pro-tip-write-flexible-kernels-grid-stride-loops/, April 22, 2013.

[5]Mark Harris. How to Access Global Memory Efficiently in CUDA C/C++ Kernels.https://developer.nvidia.com/blog/how-access-global-memory-efficiently-cuda-c-kernels/, January 7, 2013.

[6]Cyril Zeller, NVIDIA Corporation, CUDA C/C++ Basics, Supercomputing 2011 Tutorial

--

--

1

Published in TDS Archive

An archive of data science, data analytics, data engineering, machine learning, and artificial intelligence writing from the former Towards Data Science Medium publication.

Written by Raj Sengo

Software Professional, Machine Learning / AI Enthusiast. Master of Science in Software Engineering from The University of Texas, Austin.

Responses (1)

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams