Andrew Kerr, May 21, 2020
DEVELOPING CUDA KERNELS TO
PUSH TENSOR CORES TO THE
ABSOLUTE LIMIT ON NVIDIA A100
2
CUTLASS Team
Andrew Kerr, Haicheng Wu, Manish Gupta, Duane Merrill, Pradeep Ramani
Contributors
Mostafa Hagog, Timothy Costa, Alan Kaatz, John Tran, Stephen Jones, Kyrylo Perelygin, Luke
Durant, Piotr Majcher, Paul Springer, Markus Hohnerbach
Acknowledgements
Joel McCormack, Julien Demouth, Olivier Giroux, Bryce Lelbach, Cris Cecka
ACKNOWLEDGEMENTS
3
Overview
NVIDIA Ampere Architecture and CUTLASS 2.2
Tensor Cores on NVIDIA Ampere Architecture
Accelerated matrix operations
Efficient data movements for Tensor Cores
Strategies for maximizing performance
CUTLASS on NVIDIA A100
Optimal CUDA C++ templates for Tensor Cores
AGENDA
4
OVERVIEW
5
NVIDIA AMPERE ARCHITECTURE
New and Faster Tensor Core Operations

Floating-point Tensor Core operations 8x and 16x faster than F32 CUDA Cores

Integer Tensor Core operations 32x and 64x faster than F32 CUDA Cores

New IEEE double-precision Tensor Cores 2x faster than F64 CUDA Cores
Additional Data Types and Mode

Bfloat16, double, Tensor Float 32
Asynchronous copy

Copy directly into shared memory – deep software pipelines
Many additional new features – see “Inside NVIDIA Ampere Architecture”
NVIDIA A100
6
PROGRAMMING NVIDIA AMPERE ARCHITECTURE
Deep Learning and Math Libraries using Tensor Cores (with CUDA kernels under the hood)
•
cuDNN, cuBLAS, cuTENSOR, cuSOLVER, cuFFT, cuSPARSE
•
“CUDNN V8: New Advances in Deep Learning Acceleration” (GTC 2020 - S21685)
•
“How CUDA Math Libraries Can Help you Unleash the Power of the New NVIDIA A100 GPU” (GTC 2020 – S21681)
•
“Inside the Compilers, Libraries and Tools for Accelerated Computing” (GTC 2020 – S21766)
CUDA C++ Device Code
•
CUTLASS, CUDA Math API, CUB, Thrust, libcu++
CUDA device code
CUDA-accelerated math
libraries with host-side API
GPU
GPU-accelerated
application
7
PROGRAMMING NVIDIA AMPERE ARCHITECTURE
This is a talk for CUDA programmers
with CUDA C++
CUDA device code
CUDA-accelerated math
libraries with host-side API
GPU
GPU-accelerated
application
8
CUTLASS
https://github.com/NVIDIA/cutlass
CUDA C++ Templates for Deep Learning and Linear Algebra
CUDA 9.1
CUDA 10.1
CUDA 11
CUDA 9.2
CUDA 10.2
CUTLASS Preview
Release
CUTLASS 1.3 – native
NVIDIA V100 Tensor
Cores
CUTLASS 2.2 –
NVIDIA A100
CUTLASS 1.0
CUTLASS 2.0 – native
NVIDIA Turing Tensor
Cores
9
CUTLASS
CUTLASS 2.2: optimal performance on NVIDIA Ampere Architecture
•
Higher throughput Tensor Cores: more than 2x speedup for all data types
•
New floating-point types: bfloat16, Tensor Float 32, double
•
Deep software pipelines with cp.async: efficient and latency tolerant
CUTLASS 2.1
• Planar complex: complex-valued GEMMs with batching options targeting Volta and Turing Tensor Cores
• BLAS-style host side API
CUTLASS 2.0: significant refactoring using modern C++11 programming
• Efficient: particularly for Turing Tensor Cores
• Tensor Core programming model: reusable components for linear algebra kernels in CUDA
• Documentation, profiling tools, reference implementations, SDK examples, more..
What’s new?
https://github.com/NVIDIA/cutlass
10
0
100,000
200,000
300,000
400,000
500,000
600,000
700,000
800,000
900,000
1,000,000
128
1152
2176
3200
4224
5248
6272
7296
8320
9344
10368
11392
12416
13440
14464
15488
GFLOP/s
GEMM K
CUTLASS PERFORMANCE ON NVIDIA AMPERE ARCHITECTURE
0
50,000
100,000
150,000
200,000
250,000
32
544
1056
1568
2080
2592
3104
3616
4128
4640
5152
5664
6176
6688
7200
7712
GFLOP/s
GEMM K
Mixed Precision Floating Point
CUTLASS 2.2 - CUDA 11 Toolkit – NVIDIA A100
Double Precision Floating Point
Mixed Precision Integer
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
18,000
20,000
32
160
288
416
544
672
800
928
1056
1184
1312
1440
1568
1696
1824
1952
GFLOP/s
GEMM K
Tensor Core – F64
Tensor Core – BF16, F16
CUDA Core – F64
Tensor Core – TF32
CUDA Core – F32
Tensor Core – INT4
CUDA Core – INT8
Tensor Core – INT8
m=3456, n=4096
5.7x
2x
13x
7.7x
13.8x
11
TENSOR CORES ON NVIDIA
AMPERE ARCHITECTURE
12
Matrix operations: D = op(A, B) + C

Matrix multiply-add

XOR-POPC
Input Data types: A, B

half, bfloat16, Tensor Float 32, double, int8, int4, bin1
Accumulation Data Types: C, D

half, float, int32_t, double
WHAT ARE TENSOR CORES?
13
WHAT ARE TENSOR CORES?
Matrix operations: D = op(A, B) + C

Matrix multiply-add

XOR-POPC
M-by-N-by-K matrix operation

Warp-synchronous, collective operation

32 threads within warp collectively hold A, B, C, and D operands
14
NVIDIA AMPERE ARCHITECTURE - TENSOR CORE OPERATIONS
https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-instructions-mma-and-friends
PTX
Data Types
(A * B + C)
Shape
Speedup on NVIDIA A100
(vs F32 CUDA cores)
Speedup on Turing*
(vs F32 Cores)
Speedup on Volta*
(vs F32 Cores)
mma.sync.m16n8k16
mma.sync.m16n8k8
F16  * F16  + F16
F16  * F16  + F32
BF16 * BF16 + F32
16-by-8-by-16
16-by-8-by-8
16x
8x
8x
mma.sync.m16n8k8
TF32 * TF32 + F32
16-by-8-by-8
8x
N/A
N/A
mma.sync.m8n8k4
F64 * F64 + F64
8-by-8-by-4
2x
N/A
N/A
mma.sync.m16n8k32
mma.sync.m8n8k16
S8 * S8 + S32
16-by-8-by-32
8-by-8-by-16
32x
16x
N/A
mma.sync.m16n8k64
S4 * S4 + S32
16-by-8-by-64
64x
32x
N/A
mma.sync.m16n8k256
B1 ^ B1 + S32
16-by-8-by-256
256x
128x
N/A
* Instructions with equivalent functionality for Turing and Volta differ in shape from the NVIDIA Ampere Architecture in several cases.
15
Warp-wide Tensor Core operation: 8-by-8-by-128b
TENSOR CORE OPERATION: FUNDAMENTAL SHAPE
16
mma.sync.aligned
(via inline PTX)
int32_t
D[2];
uint32_t const A;
uint32_t const B;
int32_t const
C[2];
// Example targets 8-by-8-by-16 Tensor Core operation
asm(
"mma.sync.aligned.m8n8k16.row.col.s32.s8.s8.s32 "
"  { %0, %1 }, "
"    %2,       "
"    %3,       "
"  { %4, %5 }; "
:
"=r"(D[0]), "=r"(D[1])
:
"r"(A),    "r"(B),
"r"(C[0]), "r"(C[1])
);
8-by-8-by-16
S8 * S8 + S32
https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-instructions-mma-and-friends
17
Warp-wide Tensor Core operation: 16-by-8-by-128b
EXPANDING THE M DIMENSION
18
mma.sync.aligned
(via inline PTX)
float
D[4];
uint32_t const A[2];
uint32_t const B;
float   const C[4];
// Example targets 16-by-8-by-8 Tensor Core operation
asm(
"mma.sync.aligned.m16n8k8.row.col.f32.f16.f16.f32 "
"  { %0, %1, %2, %3 }, "
"  { %4, %5},          "
"    %6,               "
"  { %7, %8, %9, %10 };"
:
"=f"(D[0]), "=f"(D[1]), "=f"(D[2]), "=f"(D[3])
:
"r"(A[0]), "r"(A[1]),
"r"(B),
"f"(C[0]), "f"(C[1])
);
16-by-8-by-8
F16 * F16 + F32
https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-instructions-mma-and-friends
19
Warp-wide Tensor Core operation: 16-by-8-by-256b
EXPANDING THE K DIMENSION
20
mma.sync.aligned
(via inline PTX)
float
D[4];
uint32_t const A[4];
uint32_t const B[2];
float   const C[4];
// Example targets 16-by-8-by-32 Tensor Core operation
asm(
"mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32 "
"  { %0, %1, %2, %3 },    "
"  { %4, %5, %6, %7 },    "
"  { %8, %9 },            "
"  { %10, %11, %12, %13 };"
:
"=f"(D[0]), "=f"(D[1]), "=f"(D[2]), "=f"(D[3])
:
"r"(A[0]), "r"(A[1]), "r"(A[2]), "r"(A[3]),
"r"(B[0]), "r"(B[1]),
"f"(C[0]), "f"(C[1]), "f"(C[2]), "f"(C[3])
);
16-by-8-by-16
F16 * F16 + F32
https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-instructions-mma-and-friends
21
mma.sync.aligned
(via inline PTX)
int32_t
D[4];
uint32_t const A[4];
uint32_t const B[2];
int32_t const C[4];
// Example targets 16-by-8-by-32 Tensor Core operation
asm(
"mma.sync.aligned.m16n8k32.row.col.s32.s8.s8.s32 "
"  { %0, %1, %2, %3 },    "
"  { %4, %5, %6, %7 },    "
"  { %8, %9 },            "
"  { %10, %11, %12, %13 };"
:
"=r"(D[0]), "=r"(D[1]), "=r"(D[2]), "=r"(D[3])
:
"r"(A[0]), "r"(A[1]), "r"(A[2]), "r"(A[3]),
"r"(B[0]), "r"(B[1]),
"r"(C[0]), "r"(C[1]), "r"(C[2]), "r"(C[3])
);
16-by-8-by-32
S8 * S8 + S32
https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-instructions-mma-and-friends
22
mma.sync.aligned
(via inline PTX)
uint32_t
D[2];  // two registers needed (vs. four)
uint32_t const A[4];
uint32_t const B[2];
uint32_t const C[2];  // two registers needed (vs. four)
// Example targets 16-by-8-by-16 Tensor Core operation
asm(
"mma.sync.aligned.m16n8k16.row.col.f16.f16.f16.f16 "
"  { %0, %1},             "
"  { %2, %3, %4, %5 },    "
"  { %6, %7 },            "
"  { %8, %9 };            "
:
"=r"(D[0]), "=r"(D[1])
:
"r"(A[0]), "r"(A[1]), "r"(A[2]), "r"(A[3]),
"r"(B[0]), "r"(B[1]),
"r"(C[0]), "r"(C[1])
);
16-by-8-by-16
HALF-PRECISION : F16 * F16 + F16
https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-instructions-mma-and-friends
C[0]
C[1]
23
mma.sync.aligned
(via inline PTX)
uint64_t
D[2];   // two 64-bit accumulators
uint64_t const A;      // one 64-bit element for A operand
uint64_t const B;      // one 64-bit element for B operand
uint64_t const C[2];   // two 64-bit accumulators
// Example targets 8-by-8-by-4 Tensor Core operation
asm(
"mma.sync.aligned.m8n8k4.row.col.f64.f64.f64.f64 "
"  { %0, %1},   "
“    %2,        "
"    %3,        "
"  { %4, %5 };  "
:
"=l"(D[0]), "=l"(D[1])
:
“l"(A),
“l"(B),
“l"(C[0]), “l"(C[1])
);
8-by-8-by-4
DOUBLE-PRECISION: F64 * F64 + F64
https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-instructions-mma-and-friends
24
cutlass::arch::Mma
/// Matrix multiply-add operation
template <
/// Size of the matrix product (concept: GemmShape)
typename Shape,
/// Number of threads participating
int kThreads,
/// Data type of A elements
typename ElementA,
/// Layout of A matrix (concept: MatrixLayout)
typename LayoutA,
/// Data type of B elements
typename ElementB,
/// Layout of B matrix (concept: MatrixLayout)
typename LayoutB,
/// Element type of C matrix
typename ElementC,
/// Layout of C matrix (concept: MatrixLayout)
typename LayoutC,
/// Inner product operator
typename Operator
>
struct Mma;
m-by-n-by-k
CUTLASS: wraps PTX in template
https://github.com/NVIDIA/cutlass/blob/master/include/cutlass/arch/mma_sm80.h
25
cutlass::arch::Mma
__global__ void kernel() {
// arrays containing logical elements
Array<half_t, 8> A;
Array<half_t, 4> B;
Array< float, 4> C;
// define the appropriate matrix operation
arch::Mma< GemmShape<16, 8, 16>, 32, ... > mma;
// in-place matrix multiply-accumulate
mma(C, A, B, C);
...
}
16-by-8-by-16
CUTLASS: wraps PTX in template
https://github.com/NVIDIA/cutlass/blob/master/include/cutlass/arch/mma_sm80.h
26
EFFICIENT DATA MOVEMENT
FOR TENSOR CORES
27
CUDA example
__global__ void tensor_core_example_8x8x16(
int32_t        *D,
uint32_t const *A,
uint32_t const *B,
int32_t const  *C) {
// Compute the coordinates of accesses to A and B matrices
int outer = threadIdx.x / 4;    // m or n dimension
int inner = threadIdx.x % 4;    // k dimension
// Compute the coordinates for the accumulator matrices
int c_row = threadIdx.x / 4;
int c_col = 2 * (threadIdx.x % 4);
// Compute linear offsets into each matrix
int ab_idx = outer * 4 + inner;
int cd_idx = c_row * 8 + c_col;
// Issue Tensor Core operation
asm(
"mma.sync.aligned.m8n8k16.row.col.s32.s8.s8.s32 "
"  { %0, %1 }, "
"    %2,       "
"    %3,       "
"  { %4, %5 }; "
:
"=r"(D[cd_idx]), "=r"(D[cd_idx + 1])
:
"r"(A[ab_idx]),
"r"(B[ab_idx]),
"r"(C[cd_idx]),  "r"(C[cd_idx + 1])
);
}
HELLO WORLD: TENSOR CORES
Map each thread to coordinates of the matrix operation
Load inputs from memory
Perform the matrix operation
Store the result to memory
28
CUDA example
__global__ void tensor_core_example_8x8x16(
int32_t        *D,
uint32_t const *A,
uint32_t const *B,
int32_t const  *C) {
// Compute the coordinates of accesses to A and B matrices
int outer = threadIdx.x / 4;    // m or n dimension
int inner = threadIdx.x % 4;    // k dimension
// Compute the coordinates for the accumulator matrices
int c_row = threadIdx.x / 4;
int c_col = 2 * (threadIdx.x % 4);
// Compute linear offsets into each matrix
int ab_idx = outer * 4 + inner;
int cd_idx = c_row * 8 + c_col;
// Issue Tensor Core operation
asm(
"mma.sync.aligned.m8n8k16.row.col.s32.s8.s8.s32 "
"  { %0, %1 }, "
"    %2,       "
"    %3,       "
"  { %4, %5 }; "
:
"=r"(D[cd_idx]), "=r"(D[cd_idx + 1])
:
"r"(A[ab_idx]),
"r"(B[ab_idx]),
"r"(C[cd_idx]),  "r"(C[cd_idx + 1])
);
}
PERFORMANCE IMPLICATIONS
Load A and B inputs from memory: 2 x 4B per thread
Perform one Tensor Core operation: 2048 flops per warp
2048 flops require 256 B of loaded data
8 flops/byte
NVIDIA A100 Specifications:
•
624 TFLOP/s (INT8)
•
1.6 TB/s (HBM2)
400 flops/byte
8 flops/byte  *  1.6 TB/s   12 TFLOP/s
This kernel is global memory bandwidth limited.
29
FEEDING THE DATA PATH
Efficient storing and loading through Shared Memory
Tiled, hierarchical model: reuse data in Shared Memory and in Registers
See CUTLASS GTC 2018 talk for more details about this model.
30
FEEDING THE DATA PATH
Move data from Global Memory to Tensor Cores as efficiently as possible
• Latency-tolerant pipeline from Global Memory
• Conflict-free Shared Memory stores
• Conflict-free Shared Memory loads
31
ASYNCHRONOUS COPY: EFFICIENT PIPELINES
New NVIDIA Ampere Architecture feature: cp.async
•
Asynchronous copy directly from Global to Shared Memory
•
See “Inside the NVIDIA Ampere Architecture” for more details (GTC 2020 – S21730)
Enables efficient software pipelines
•
Minimizes data movement: L2 L1 RF SMEM becomes L2 SMEM
•
Saves registers: RF no longer needed to hold the results of long-latency load instructions
•
Indirection: fetch several stages in advance for greater latency tolerance
Committed
Stage
SMEM write
pointer
Copies in flight
Circular buffer in Shared Memory
cp.async
cp.async
cp.async
ld.shared
32
FEEDING THE DATA PATH
Move data from Global Memory to Tensor Cores as efficiently as possible
• Latency-tolerant pipeline from Global Memory
• Conflict-free Shared Memory stores
• Conflict-free Shared Memory loads
33
GLOBAL MEMORY TO TENSOR CORES
Global Memory
Shared Memory
Tensor Cores
cp.async
M dimension
K dimension
34
LDMATRIX: FETCH TENSOR CORE OPERANDS
PTX instruction to load a matrix from Shared Memory
Shared Memory
Tensor Cores
Each thread supplies a pointer to 128b row of data in Shared Memory
Each 128b row is broadcast to groups of four threads
(potentially different threads than the one supplying the pointer)
Data matches arrangement of inputs to Tensor Core operations
35
LDMATRIX: PTX INSTRUCTION
Each thread supplies a pointer to 128b row of data in Shared Memory
Each 128b row is broadcast to groups of four threads
(potentially different threads than the one supplying the pointer)
Data matches arrangement of inputs to Tensor Core operations
PTX instruction to load a matrix from SMEM
// Inline PTX assembly for ldmatrix
uint32_t
R[4];
uint32_t
smem_ptr;
asm volatile (
"ldmatrix.sync.aligned.x4.m8n8.shared.b16 "
"{%0, %1, %2, %3}, [%4];                  "
:
"=r"(R[0]), "=r"(R[1]), "=r"(R[2]), "=r"(R[3])
:
"r"(smem_ptr)
);
36
GLOBAL MEMORY TO TENSOR CORES
Tensor Cores
ldmatrix
cp.async
Global Memory
Shared Memory
37
NVIDIA AMPERE ARCHITECTURE – SHARED MEMORY BANK TIMING
Bank conflicts between threads in the same phase
4B words are accessed in 1 phase
8B words are accessed in 2 phases:
• Process addresses of the first 16 threads in a warp
• Process addresses of the second 16 threads in a warp
16B words are accessed in 4 phases:
• Each phase processes 8 consecutive threads of a warp
Slide borrowed from: Guillaume Thomas-Collignon and Paulius Micikevicius. "Volta Architecture and performance optimization.”
GTC 2018.
http://on-demand.gputechconf.com/gtc/2018/presentation/s81006-volta-architecture-and-performance-optimization.pdf
128 bit access size
Phase 0:  T0 .. T7
Phase 1:  T8 .. T15
Phase 2: T16 .. T23
Phase 3: T24 .. T31
38
GLOBAL MEMORY TO TENSOR CORES
Global Memory
Shared Memory
Registers
Bank conflict on either store or load
from Shared Memory
ldmatrix
cp.async
39
GLOBAL TO SHARED MEMORY
Load
(128 bits per thread)
Store
(128 bits per thread)
Permuted Shared Memory layout
XOR function maps thread index to Shared
Memory location
40
GLOBAL TO SHARED MEMORY
Phase 0:  T0 .. T7
Phase 1:  T8 .. T15
Phase 2: T16 .. T23
Phase 3: T24 .. T31
Load
(128 bits per thread)
Store
(128 bits per thread)
41
GLOBAL TO SHARED MEMORY
Phase 0:  T0 .. T7
Phase 1:  T8 .. T15
Phase 2: T16 .. T23
Phase 3: T24 .. T31
Load
(128 bits per thread)
Store
(128 bits per thread)
42
GLOBAL TO SHARED MEMORY
Phase 0:  T0 .. T7
Phase 1:  T8 .. T15
Phase 2: T16 .. T23
Phase 3: T24 .. T31
Load
(128 bits per thread)
Store
(128 bits per thread)
43
GLOBAL TO SHARED MEMORY
Phase 0:  T0 .. T7
Phase 1:  T8 .. T15
Phase 2: T16 .. T23
Phase 3: T24 .. T31
Load
(128 bits per thread)
Store
(128 bits per thread)
44
FEEDING THE DATA PATH
Move data from Global Memory to Tensor Cores as efficiently as possible
• Latency-tolerant pipeline from Global Memory
• Conflict-free Shared Memory stores
• Conflict-free Shared Memory loads
45
LOADING FROM SHARED MEMORY TO REGISTERS
46
LOADING FROM SHARED MEMORY TO REGISTERS
47
LOADING FROM SHARED MEMORY TO REGISTERS
48
LOADING FROM SHARED MEMORY TO REGISTERS
49
ADVANCING TO NEXT K GROUP
K=16 .. 31
K=0 ..15
50
ADVANCING TO NEXT K GROUP
smem_ptr = row_idx * 8 + column_idx;
smem_ptr = smem_ptr ^ 2;
K=0..15
K=16..31
51
LOADING FROM SHARED MEMORY TO REGISTERS
Phase 0
K=16..31
52
LOADING FROM SHARED MEMORY TO REGISTERS
Phase 1
K=16..31
53
LOADING FROM SHARED MEMORY TO REGISTERS
Phase 2
K=16..31
54
LOADING FROM SHARED MEMORY TO REGISTERS
Phase 3
K=16..31
55
CUTLASS
CUDA C++ Templates as an Optimal Abstraction Layer for Tensor Cores
• Latency-tolerant pipeline from Global Memory
• Conflict-free Shared Memory stores
• Conflict-free Shared Memory loads
56
CUTLASS: OPTIMAL ABSTRACTION FOR TENSOR CORES
using Mma = cutlass::gemm::warp::DefaultMmaTensorOp<
GemmShape<64, 64, 16>,
half_t, LayoutA,
// GEMM A operand
half_t, LayoutB,
// GEMM B operand
float, RowMajor
// GEMM C operand
>;
__shared__ ElementA smem_buffer_A[Mma::Shape::kM * GemmK];
__shared__ ElementB smem_buffer_B[Mma::Shape::kN * GemmK];
// Construct iterators into SMEM tiles
Mma::IteratorA iter_A({smem_buffer_A, lda}, thread_id);
Mma::IteratorB iter_B({smem_buffer_B, ldb}, thread_id);
Mma::FragmentA frag_A;
Mma::FragmentB frag_B;
Mma::FragmentC accum;
Mma mma;
accum.clear();
#pragma unroll 1
for (int k = 0; k < GemmK; k += Mma::Shape::kK) {
iter_A.load(frag_A);  // Load fragments from A and B matrices
iter_B.load(frag_B);
++iter_A; ++iter_B;   // Advance along GEMM K to next tile in A
//   and B matrices
// Compute matrix product
mma(accum, frag_A, frag_B, accum);
}
Shared Memory
Tensor Cores
Warp-level
matrix multiply
57
CUTLASS: OPTIMAL ABSTRACTION FOR TENSOR CORES
using Mma = cutlass::gemm::warp::DefaultMmaTensorOp<
GemmShape<64, 64, 16>,
half_t, LayoutA,
// GEMM A operand
half_t, LayoutB,
// GEMM B operand
float, RowMajor
// GEMM C operand
>;
__shared__ ElementA smem_buffer_A[Mma::Shape::kM * GemmK];
__shared__ ElementB smem_buffer_B[Mma::Shape::kN * GemmK];
// Construct iterators into SMEM tiles
Mma::IteratorA iter_A({smem_buffer_A, lda}, thread_id);
Mma::IteratorB iter_B({smem_buffer_B, ldb}, thread_id);
Mma::FragmentA frag_A;
Mma::FragmentB frag_B;
Mma::FragmentC accum;
Mma mma;
accum.clear();
#pragma unroll 1
for (int k = 0; k < GemmK; k += Mma::Shape::kK) {
iter_A.load(frag_A);  // Load fragments from A and B matrices
iter_B.load(frag_B);
++iter_A; ++iter_B;   // Advance along GEMM K to next tile in A
//   and B matrices
// Compute matrix product
mma(accum, frag_A, frag_B, accum);
}
Tile Iterator Constructors:
Initialize pointers into permuted Shared Memory buffers
Fragments:
Register-backed arrays holding each thread’s data
Warp-level matrix multiply:
Decomposes a large matrix multiply into Tensor Core operations
Tile Iterator:
load() - Fetches data from permuted Shared Memory buffers
operator++()  - advances to the next logical matrix in SMEM
58
CUTLASS ON NVIDIA A100
59
cuBLAS
CUTLASS
99%
99%
98%
99%
95%
93%
96%
95%
98%
97%
98%
97%
94%
90%
97%
99%
90%
90%
90%
90%
80%
83%
80%
80%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
NN
NT
TN
TT
NN
NT
TN
TT
NN
NT
TN
TT
NN
NT
TN
TT
NN
NT
TN
TT
NN
NT
TN
TT
DGEMM
IGEMM
SGEMM
TensorOp (f16)
TensorOp (f32)
TensorOp (TF32)
CUTLASS RELATIVE PERFORMANCE TO CUBLAS
CUTLASS 2.2 – CUDA 11 Toolkit – NVIDIA A100
60
CUTLASS RELATIVE PERFORMANCE TO CUBLAS
cuBLAS
CUTLASS 2.2 – CUDA 11 Toolkit – Three generations of GPU architectures
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
NN
NT
TN
TT
NN
NT
TN
TT
NN
NT
TN
TT
NN
NT
TN
TT
NN
NT
TN
TT
NN
NT
TN
TT
DGEMM
IGEMM
SGEMM
TensorOp (f16)
TensorOp (f32)
TensorOp (TF32)
2080Ti
A100
TitanV
CUTLASS
61
0
50,000
100,000
150,000
200,000
250,000
64
160
256
352
448
544
640
736
832
928
1024
1120
1216
1312
1408
1504
1600
1696
1792
1888
1984
2080
2176
2272
2368
2464
2560
2656
2752
2848
2944
3040
3136
3232
3328
3424
3520
3616
3712
3808
3904
4000
4096
GFLOP/s
GEMM K
ARBITRARY PROBLEM SIZE
CUTLASS Templates Cover the Design Space
128b alignment
64b alignment
32b alignment
16b alignment
CUDA 10.2 and
before
CUTLASS 2.2 – NVIDIA A100 - Tensor Cores: F16 * F16 + F32
62
CONCLUSION
63
CONCLUSION: NVIDIA A100 IS FAST AND PROGRAMMABLE
Tensor Cores on NVIDIA A100 in CUDA
• Order of magnitude speedup for matrix computations
• Programmable in CUDA via mma.sync with zero overhead
• Kernel design can avoid memory bottlenecks
• CUDA 11 Toolkit capable of near-peak performance
CUTLASS 2.2: May 2020
•
Open source CUDA C++ template library for CUDA development
•
Reusable building blocks for utilizing Tensor Cores on NVIDIA GPUs
•
Near-optimal performance on NVIDIA Ampere Architecture
Try it out! https://github.com/NVIDIA/cutlass
64
REFERENCES
NVIDIA Ampere Architecture:
“Inside the NVIDIA Ampere Architecture” (GTC 2020 – S21730)
“NVIDIA Ampere Architecture In-Depth” (blog post)
“CUDA New Features and Beyond” (GTC 2020 – S21760)
“Tensor Core Performance on NVIDIA GPUs” (GTC 2020 – S21929)
“Inside the Compilers, Libraries and Tools for Accelerated Computing” (GTC 2020 - S21766)
CUTLASS
https://github.com/NVIDIA/cutlass (open source software, New BSD license)
GTC 2018 and GTC 2019 talks: GEMM structure and Volta Tensor Cores
CUTLASS Parallel For All blog post


3D Game Engine Programming

Helping you build your dream game engine

Main menu

Post navigation

CUDA Thread Execution Model

Grid of Thread Blocks

Contents

GPU Architecture

To understand the thread execution model for modern GPU’s, we must first make an analysis of the GPU compute architecture. In this article I will focus on the Fermi compute architecture found in modern GPU’s (GTX 580).

Overview of the Fermi Architecture

A Fermi GPU consists of 512 CUDA cores. These 512 CUDA cores are split across 16 Streaming Multiprocessors (SM) each SM consisting of 32 CUDA cores. The GPU has 6 64-bit memory partitions supporting up to 6 GB of GDDR5 DRAM memory.

Fermi Arcitecture

Each streaming multiprocessor (SM) has 32 cuda cores. Each CUDA core consists of an integer arithmetic logic unit (ALU) and a floating point unit (FPU).

Fermi Streaming Multiprocessor (SM)

The SM has 16 load/store units allowing source and destination addresses to be calculated for sixteen threads per clock.

Each SM also has four Special Function Units (SFU) that execute transcendental instructions such as sin, cosine, reciprocal, and square root.

CUDA Threads

Now that we’ve seen the specific architecture of a Fermi GPU, let’s analyze the more general CUDA thread execution model.

Each kernel function is executed in a grid of threads. This grid is divided into blocks also known as thread blocks and each block is further divided into threads.

Cuda Execution Model

In the image above we see that this example grid is divided into nine thread blocks (3×3), each thread block consists of 9 threads (3×3) for a total of 81 threads for the kernel grid.

This image only shows 2-dimensional grid, but if the graphics device supports compute capability 2.0, then the grid of thread blocks can actually be partitioned into 1, 2 or 3 dimensions, otherwise if the device supports compute capability 1.x, then thread blocks can be partitioned into 1, or 2 dimensions (in this case, then the 3rd dimension should always be set to 1).

The thread block is partitioned into individual threads and for all compute capabilities, threads can be partitioned into 1, 2, or 3 dimensions. The maximum number of threads that can be assigned to a thread block is 512 for devices with compute capability 1.x and 1024 threads for devices that support compute capability 2.0.

The number of blocks within a gird can be determined within a kernel by using the built-in variable gridDim and the number of threads within a block can be determined by using the built-in variable blockDim.

A thread block is uniquely identified in a kernel function by using the built-in variable blockIdx and a thread within a block is uniquely identified in a kernel function by using the built-in variable threadIdx.

The built-in variables gridDim, blockDim, blockIdx, and threadIdx are each 3-component structs with members x, y, z.

With a 1-D kernel, the unique thread ID within a block is the same as the x component of the threadIdx variable.

and the unique block ID within a grid is the same as the x component of the blockIdx variable:

To determine the unique thread ID in a 2-D block, you would use the following formula:

and to determine the unique block ID within a 2-D grid, you would use the following formula:

I’ll leave it as an exercise for the reader to determine the formula to compute the unique thread ID and block ID in a 3D grid.

Matrix Addition Example

Let’s take a look at an example kernel that one might execute.

Let’s assume we want to implement a kernel function that adds two matrices and stores the result in a 3rd.

The general formula for matrix addition is:

That is, the sum of matrix A and matrix B is the sum of the components of matrix A and matrix B.

Let’s first write the host version of this method that we would execute on the CPU.

This is a pretty standard method that loops through the rows and columns of a matrix and adds the components and stores the results in a 3rd. Now let’s see how we might execute this kernel on the GPU using CUDA.

First, we need to think of the problem domain. I this case, the domain is trivial: it is the components of a matrix. Since we are operating on 2-D arrays, it seems reasonable to split our domain into two dimensions; one for the rows, and another for the columns of the matrices.

We will assume that we are working on square matrices. This simplifies the problem but mathematically matrix addition only requires that the two matrices have the same number of rows and columns but does not have the requirement that the matrices must be square.

Since we know that a kernel is limited to 512 threads/block with compute capability 1.x and 1024 threads/block with compute capability 2.0, then we know we can split our job into square thread blocks each consisting of 16×16 threads (256 threads per block) with compute capability 1.x and 32×32 threads (1024 threads per block) with compute capability 2.0.

If we limit the size of our matrix to no larger than 16×16, then we only need a single block to compute the matrix sum and our kernel execution configuration might look something like this:

In this simple case, the kernel grid consists of only a single block with matrixRank x matrixRank threads.

However, if we want to sum matrices larger than 512 components, then we must split our problem domain into smaller groups that can be processed in multiple blocks.

Let’s assume that we want to limit our blocks to execute in 16×16 (256) threads. We can determine the number of blocks that will be required to operate on the entire array by dividing the size of the matrix dimension by the maximum number of threads per block and round-up to the nearest whole number:

And we can determine the number of threads per block by dividing the size of the matrix dimension by the number of blocks and round-up to the nearest whole number:

So for example, for a 4×4 matrix, we would get

and the number of threads is computed as:

resulting in a 1×1 grid of 4×4 thread blocks for a total of 16 threads.

Another example a 512×512 matirx, we would get:

and the number of threads is computed as:

resulting in a 32×32 grid of 16×16 thread blocks for a total of 262,144 threads.

The host code to setup the kernel granularity might look like this:

The Matrix Addition Kernel Function

On the device, one kernel function is created for every thread in the problem domain (the matrix elements). We can use the built-in variables gridDim, blockDim, blockIdx, and threadIdx, to identify the current matrix element that the current kernel is operating on.

If we assume we have a 9×9 matrix and we split the problem domain into 3×3 blocks each consisting of 3×3 threads as shown in the CUDA Grid below, then we could compute the ith column and the jth row of the matrix with the following formula:

So for thread (0,0) of block (1,1) of our 9×9 matrix, we would get:

for the column and:

for the row.

The index into the 1-D buffer that store the matrix is then computed as:

and substituting gives:

Which is the correct element in the matrix. This solution assumes we are accessing the matrix in row-major order.

CUDA Grid Example

Let’s see how we might implement this in the kernel.

On line 3, and 4 we compute the column and row of the matrix we are operating on using the formulas shown earlier.

On line 6, the 1-d index in the matrix array is computed based on the size of a single dimension of the square matris.

We must be careful that we don’t try to read or write out of the bounds of the matrix. This might happen if the size of the matrix does not fit nicely into the size of the CUDA grid (in the case of matrices whose size is not evenly divisible by 16) To protect the read and write operation, on line 7 we must check that the computed index does not exceed the size of our array.

Thread Synchronization

CUDA provides a synchronization barrier for all threads in a block through the __syncthreads() method. A practical example of thread synchronization will be shown in a later article about optimization a CUDA kernel, but for now it’s only important that you know this functionality exists.

Thread synchronization is only possible across all threads in a block but not across all threads running in the grid. By not allowing threads across blocks to be synchronized, CUDA enables multiple blocks to be executed on other streaming multiprocessors (SM) in any order. The queue of blocks can be distributed to any SM without having to wait for blocks from another SM to be complete. This allows the CUDA enabled applications to scale across platforms that have more SM at it’s disposal, executing more blocks concurrently than another platforms with less SM’s.

Thread synchronization follows strict synchronization rules. All threads in a block must hit the synchronization point or none of them must hit synchronization point.

Give the following code block:

will cause the threads in a block to wait indefinitely for each other because the two occurrences of __syncthreads are considered separate synchronization points and all threads of the same block must hit the same synchronization point, or all of them must not hit it.

Thread Assignment

When a kernel is invoked, the CUDA runtime will distribute the blocks across the SM’s on the device. A maximum of 8 blocks (irrelevant of platform) will be assigned to each SM as long as there are enough resources (registers, shared memory, and threads) to execute all the blocks. In the case where there are not enough resources on the SM, then the CUDA runtime will automatically assign less blocks per SM until the resource usage is below the maximum per SM.

The total number of blocks that can be executed concurrently is dependent on the device. In the case of the Fermi architecture discussed earlier, a total of 16 SM’s can concurrently handle 8 blocks for a total of 128 blocks executing concurrently on the device.

Because the Fermi architecture support compute compatibility 2.0, we can create thread blocks consisting of at most 1024 threads, then the Fermi device can technically support 131,072 threads residing in the SM’s for execution. This does not mean that every clock tick the devices is executing 131,072 instruction simultaneously. In order to understand how the blocks are actually executed on the device, we must look one step further to see how the threads of a block are actually scheduled on the SM’s.

Thread Scheduling

When a block is assigned to a SM, it is further divided into groups of 32 threads called a warp. Warp scheduling is different depending on the platform, but if we take a look at the Fermi architecture, we see that a single SM consists of 32 CUDA cores (or streaming processor) – two groups of 16 per SM.

Each SM in the Fermi architecture (see Fermi architecture image above) features two warp schedulers allowing two warps to be issued and executed concurrently. Fermi’s dual-warp scheduler selects two warps and issues one instruction from each warp to a group of sixteen cores, sixteen load/store units, or four special function units (SFU’s).

Most instructions can be dual-issued; two integer instructions, two floating point instructions, or a mix of integer, floating point, load, store, and SFU instructions can be issued concurrently.

Fermi - Dual Warp Scheduler

You might be wondering why it would be useful to schedule 8 blocks of a maximum of 1024 threads if the SM only has 32 SP’s? The answer is that each instruction of a kernel may require more than a few clock cycles to execute (for example, an instruction to read from global memory will require multiple clock cycles). Any instruction that requires multiple clock cycles to execute incurs latency. The latency of long-running instructions can be hidden by executing instructions from other warps while waiting for the result of the previous warp. This technique of filling the latency of expensive operations with work from other threads is often called latency hiding.

Thread Divergence

It is reasonable to imagine that your CUDA program contains flow-control statements like if-then-else, switch, while loops, or for loops.  Whenever you introduce these flow-control statements in your code, you also introduce the possibility of thread divergence.  It is important to be aware of the consequence of thread divergence and also to understand how you can minimize the negative impact of divergence.

Thread divergence occurs when some threads in a warp follow a different execution path than others.  Let’s take the following code block as an example:

Then our flow control and thread divergence would look something like this:

Thread Divergence

As you can see from this example, the even numbered threads in each block will execute PathA while the odd numbered threads in the block will execute PathB.  This is pretty much the worst-case scenario for simple divergence example.

Both PathA and PathB cannot be executed concurrently on all threads because their execution paths are different.  Only the threads that execute the exact same execution path can run concurrently so the total running time of the warp is the sum of the execution time of both PathA and PathB.

In this example, the threads in the warp that execute PathA are activated if the condition is true and all the other threads are deactivated.  Then, in another pass, all the threads that execute PathB are activated if the condition is false are activated and the other threads are deactivated.  This means that to resolve this condition requires 2-passes to be executed for a single warp.

The overhead of having the warp execute both PathA and PathB can be eliminated if the programmer  takes careful consideration when writing the kernel.  If possible, all threads of a block (since warps can’t span thread blocks) should execute the same execution path.  This way you guarantee that all threads in a warp will execute the same execution path and there will be no thread divergence within a block.

Exercise

If a device supports compute capability 1.3 then it can have blocks with a maximum of 512 threads/block and 8 blocks/SM can be scheduled concurrently. Each SM can schedule groups of 32-thread units called warps. The maximum number of resident warps per SM in a device that supports compute capability 1.3 is 32 and the maximum number of resident threads per SM is 1024.

Q. What would be the ideal block granularity to compute the product of two 2-D matrices of size 1024 x 1024?

A. To answer this question, let’s analyze each choice and give pros and cons for each one.

4×4: If we decide to split our domain into 4×4 thread blocks, then we have 16 threads per block. In order to fully occupy the SM that can support 1024 threads per SM, we would need 1024/16 = 64 blocks but the SM can only schedule 8 blocks/SM so each SM would be scheduled with 8 blocks each having 16 threads which is 128 threads/SM. When divided into warps, we only have 4 warps scheduled per SM out of a total of 32 which gives only 12.5% occupancy.

8×8: We have the same problem here as we did with the 4×4 thread block granularity except not as severe. With 8×8 thread blocks, we get 64 threads per block. For a SM that can support 1024 threads per SM, we would need 1024/64 = 16 blocks but since we are limited to 8 blocks maximum per SM, we can only execute 8×64 = 512 threads/SM. When split into warps of 32-threads each, we get 512/32 = 16 warps scheduled per SM from a possible total 32 warps. This give only 50% occupancy.

16×16: A 16×16 thread block gives 256 threads/block. With a maximum thread limit per SM of 1024, we get 1024/256 = 4 blocks/SM. This is within the 8 block limit so 4 blocks, each of 256 threads can be scheduled on one SM. With 4 blocks each with 256 threads, we get a total of 1024 threads. The threads are further split into warps of 32 threads each for a total of 32 warps. Since the device can support 32 warps/SM we have achieved 100% occupancy.

32×32: This option is not even an option since a 32×32 thread block produces a single block with 1024 threads. As stated earlier, we are limited to 512 threads per block with compute capability 1.3 so our kernel wouldn’t even run.

So the best choice for this problem domain would be to invoke a kernel with block size 16×16.

Conclusion

In this article, I discussed the architecture of a CUDA enabled GPU, in particular the Fermi architecture.  I also showed how a kernel function is scheduled on the GPU and how the warp scheduler executes instructions from different warps in order to minimize the amount of noticeable latency between kernel instructions.

References

2 thoughts on “CUDA Thread Execution Model”

I have a question about this exercise.
There is matrix dim = 1024 and 1024 threads per SM.
Where you use dim value and where threads number on these calculate?
It’s looks like dim value is never used…

Btw. This blog is awesome 🙂

The size of the matrix is not really important when you are trying to determine thread occupancy.  The only requirement is that the matrix should be large enough to keep all the SM’s busy.  The 1024×1024 matrix size ensures that we have enough threads to process.

The point of the exercise is to show that if you make the block sizes too small, you will not be able to maintain full occupancy on the GPU and if you make the block sizes too big, it won’t even run.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Comment *

Name *

Email *

Website

Save my name, email, and website in this browser for the next time I comment.

Δ

This site uses Akismet to reduce spam. Learn how your comment data is processed.


Home

Articles

Developers FAQ

Cuda developers questions

Optimize Your First CUDA Code with Best Practices

Published on
        17 February 2025
 by Cătălina Mărcuță & MoldStud Research Team

Optimize Your First CUDA Code with Best Practices

Discover essential best practices to optimize your first CUDA code, enhancing performance and efficiency for new developers in GPU programming.

In today's technology-driven landscape, the demand for high-performance computing is ever-increasing. As programmers embark on their journey into parallel programming, understanding the nuances of coding for graphical processing units can be pivotal. This area of development isn't just about writing functional scripts; it's about ensuring that those scripts run as effectively as possible. With the right approach, developers can unlock the full potential of their hardware.

Many novice programmers may feel overwhelmed by the complexities involved in harnessing the power of GPUs. Yet, with a strategic mindset, they can streamline their development process and achieve remarkable results. Simple mistakes can often lead to significant performance bottlenecks. Therefore, acquiring essential knowledge before diving in can make all the difference.

Consider the importance of memory management. In GPU programming, the way data is handled can drastically affect execution speed. Efficiently utilizing shared and global memory, for instance, can lead to noticeable improvements. Not to mention, understanding how to minimize memory transfers between the CPU and GPU is crucial.

Additionally, grasping the concepts of parallelism can significantly enhance program performance. When developers learn to break tasks into smaller chunks that can be processed simultaneously, they harness the true power of modern GPUs. This is where the real magic happens; programmers can take advantage of hundreds or even thousands of processing cores to perform calculations in tandem.

In this article, we will delve into various techniques and guidelines that can aid in achieving stellar performance in GPU applications. We will present valuable insights, practical tips, and code examples to illuminate your path. Remember, understanding the tools at your disposal is key to becoming a proficient programmer in this exciting field.

Optimizing Your First CUDA Code

Writing efficient parallel algorithms can significantly impact application performance. Every detail in your implementation can play a role in maximizing execution speed. Even small optimizations can result in noticeable improvements. Understanding memory access patterns and kernel execution is crucial.

Consider the following strategies to enhance performance:

Incorporating these tips can lead to substantial enhancements in computational performance, as many developers discover through rigorous testing and profiling of their applications. For instance, using shared memory efficiently can sometimes reduce access times by several orders of magnitude.

The following example illustrates the use of shared memory:

Testing various configurations and using profiling tools can reveal performance bottlenecks. Tools such as NVIDIA Visual Profiler can provide insights into your kernel's execution characteristics. Analyzing memory access patterns and understanding occupancy levels are vital steps toward achieving high efficiency.

As you advance in your journey, remember that connecting with experienced developers can provide invaluable insights. Consider collaborating with professionals, such as those you can hire vaadin developers, to enhance your knowledge and improve your coding practices.

Fundamentals of CUDA Programming

Understanding the basic principles of parallel programming is crucial. Harnessing the power of GPUs can significantly enhance performance. This section delves into the core concepts that underpin GPU programming. Starting with architecture, it's vital to grasp how threads, blocks, and grids work together. Each element plays a unique role in executing tasks efficiently and concurrently.

Threads are the smallest units of execution. They are grouped into blocks, which are then organized into grids. This hierarchical structure allows for scalable parallelism. Knowing how to structure these components can lead to better performance. For instance, when launching a kernel, determining the appropriate number of threads and blocks is essential.

Memory management is another critical area to explore. GPUs have different types of memory, such as global, shared, and registers. Each type offers varying access speeds and storage capacities. Effective memory use can drastically affect processing times. To illustrate, consider the following memory types and their characteristics:

Additionally, understanding how to manage data transfers between the host and GPU is fundamental for performance tuning. Often, the bottleneck lies in data transfer times rather than computation. Therefore, minimizing these transfers can lead to more efficient execution. By leveraging asynchronous memory copies and streams, developers can overlap computation and data transfer, significantly enhancing performance.

In conclusion, mastering these fundamental principles sets the stage for developing efficient applications. Familiarity with thread management, memory types, and data transfers is essential. Engaging with these concepts ensures a solid foundation for more advanced techniques. As programmers become more adept, they'll find new ways to exploit the parallel nature of GPUs. By doing so, the benefits of GPU computing become increasingly accessible and impactful.

Understanding the CUDA Architecture

Having a grasp of the architecture behind parallel computing is crucial. This knowledge enables developers to write more efficient applications. The CUDA framework is designed to facilitate this by utilizing the power of GPUs. Understanding its fundamental components can significantly enhance performance. A well-structured architecture can lead to improved execution speeds and lower latency.

The architecture is built around several key elements. These include Streaming Multiprocessors (SMs) and CUDA cores. SMs are responsible for executing threads simultaneously. Each SM contains numerous CUDA cores, which handle individual tasks. The parallel execution model allows for vast acceleration in computation.

Furthermore, memory hierarchy plays a vital role. There are various types of memory, including global, shared, and local. Global memory is accessible to all threads but has higher latency. Shared memory, on the other hand, allows threads within the same block to communicate more efficiently. Understanding how to utilize these memory types can lead to significant performance gains.

Here’s a quick comparison of the different memory types used in the architecture:

Effective implementation of parallel algorithms relies on understanding how to effectively manage these memory types. The ability to arrange threads and blocks properly is essential for maximizing resource utilization. Moreover, the synchronizing of thread execution can greatly impact computational efficiency. For developers looking to enhance their project outcomes, knowledge of these architectural elements is indispensable.

Consider the comparison of execution times across different platforms. According to recent statistics, applications utilizing GPU acceleration can achieve speeds up to 100 times faster than those using traditional processors. This difference can be the deciding factor in performance-sensitive applications.

As you delve deeper into the intricacies of this architecture, it may also be beneficial to seek external expertise. For example, many businesses find it advantageous to hire wordpress plugin developers to enhance their capabilities. By collaborating with experienced professionals, you can unlock the full potential of parallel computing.

Memory Hierarchy and Its Importance

Understanding the structure of memory in computing systems is crucial for efficient performance. Memory hierarchy significantly affects how data is accessed and processed. Each level of memory offers different speeds, sizes, and costs. By recognizing these differences, one can make informed decisions about data placement and access patterns.

At the top of the hierarchy lies the registers, which provide the fastest access speeds. Next, there are shared memory and local memory, both of which play vital roles in managing data during processing. Global memory follows, offering larger storage but with higher latency. The hierarchy continues with device memory and host memory, where communication speed varies greatly.

Efficient use of memory can enhance overall performance. For instance, placing frequently accessed data in shared memory reduces access times and improves computational efficiency. Furthermore, minimizing the use of global memory can lead to significant performance gains, especially in data-intensive applications.

For example, consider optimizing matrix multiplication, a common operation in many applications. By storing submatrices in shared memory, one can dramatically reduce the number of expensive global memory accesses. This approach can lead to performance improvements, sometimes exceeding three times the speed of naive implementations.

Ultimately, grasping memory hierarchy enables developers to write applications that leverage available resources more effectively, paving the way for achieving optimal performance in applications. It's essential to regularly profile memory usage and performance metrics to identify bottlenecks and optimize data flow.

In summary, a deep understanding of memory hierarchy is vital for any software engineer aiming to enhance performance in their applications. By employing specific strategies centered on memory usage, developers can ensure efficient data access and significantly lower execution times.

Thread Management Strategies

Thread management plays a critical role in achieving peak performance when working with parallel processing. Properly handling threads can significantly influence computation speed and resource utilization. A well-structured approach ensures that all hardware capabilities are effectively leveraged. Understanding the nuances of how threads interact with one another and with memory can lead to impressive enhancements in application efficiency.

When deploying kernels, maintaining a balanced workload across threads is essential. Ensure each thread has roughly equal tasks to prevent bottlenecks. One common method is to utilize a grid-stride loop. It allows threads to execute tasks in a staggered manner, which can lead to better occupancy rates.

__global__ void kernelFunction(int *data) { int idx = blockIdx.x * blockDim.x + threadIdx.x; for (int i = idx; i < N; i += blockDim.x * gridDim.x) { data[i] = performComputation(data[i]); } }

Adopting the grid-stride technique helps ensure that even if one thread finishes early, others can continue processing, thus maximizing hardware utilization. Furthermore, effectively managing thread divergence is crucial for maintaining consistent performance. Divergence occurs when threads within the same warp take different execution paths, leading to inefficient use of resources.

To minimize divergence, it's advisable to structure code in such a way that similar threads take the same execution path whenever possible. Conditional statements should be carefully placed to avoid unnecessary branching. Tools for profiling can help identify divergence hotspots within your application. Eliminate or minimize these to bolster efficiency.

Another technique is to experiment with different block sizes and configuration parameters based on the specific hardware being used. For instance, using optimally sized blocks can enhance the occupancy of the streaming multiprocessors (SMs), thus improving overall performance. Testing various configurations can reveal the best balance for your specific application.

In summary, effective thread management encompasses various strategies, including workload balancing, minimizing divergence, and optimizing block sizes. Leveraging these techniques will lead to more efficient execution and better resource utilization, driving performance enhancements in parallel processing tasks.

Best Practices for Performance Enhancement

Enhancing performance in GPU programming requires a thoughtful approach. Simple optimizations can yield significant speed-ups. Fine-tuning algorithms and understanding hardware capabilities is crucial. Certain strategies can maximize throughput while minimizing latency. Let's explore effective methods to elevate performance levels.

One fundamental rule is to minimize data transfers between host and device. Memory bandwidth is often the bottleneck in GPU applications. Keeping data on the device as long as possible can lead to substantial speed increases. Batch processing of data can mitigate transfer costs and make operations more efficient. For instance, performing multiple computations before sending data back can help.

Another critical element involves leveraging the architecture for better execution. Proper utilization of shared memory can improve access times and reduce global memory traffic, as shared memory is significantly faster. Consider breaking down large operations into smaller tasks that can efficiently use shared memory. A well-structured kernel design can help. For example:

Implementing concurrent execution also plays a pivotal role in boosting performance. By overlapping computation and data transfers, applications can make effective use of the GPU’s capabilities. Managing kernels and memory copies to run simultaneously can help utilize resources efficiently. For example, employing streams allows developers to orchestrate multiple tasks.

Lastly, always profile and benchmark performance regularly. Identifying bottlenecks through profiling tools gives insight into where improvements can be made. Fine-tuning based on real data ensures a targeted enhancement approach. Integrating these strategies will lead to more efficient and high-performing applications.

Efficient Memory Usage and Allocation

Memory management is critical in any parallel computing environment. Proper allocation and usage can significantly impact performance and resource utilization. It's essential to consider various aspects when dealing with memory in GPU programming. Efficient memory strategies can lead to better throughput and lower execution times. This section covers the key elements of memory handling to maximize efficiency.

Understanding the different types of memory available on the GPU is fundamental. Global memory is the most extensive but also the slowest. Shared memory, on the other hand, offers much faster access but is limited in size. Using shared memory effectively can reduce latency significantly, as threads within the same block can access it quickly. In scenarios where data sharing among threads occurs frequently, shared memory becomes invaluable.

Additionally, memory allocation strategies play a significant role. Allocating memory during kernel execution can lead to performance penalties. Instead, it's advisable to allocate all necessary memory before launching the kernel. This practice minimizes overhead and helps in maintaining a smoother execution flow.

Consider the following example to illustrate effective memory allocation:

Here, memory for a floating-point array is allocated on the device before the kernel execution. Preallocation reduces the risk of fragmentation and optimizes memory usage.

Moreover, when transferring data between host and device, batch processing can enhance efficiency. Instead of multiple small transfers, larger, consolidated transfers are preferred. This approach can help to leverage the high throughput available on the PCIe bus, reducing the total time spent waiting for memory operations to complete.

Implementing memory coalescing techniques can also lead to substantial performance gains. This involves arranging data accesses in a manner that maximizes memory throughput. When consecutive threads access consecutive memory addresses, the GPU can efficiently handle these requests, ensuring better resource use.

In conclusion, effective management of memory usage and allocation is crucial for optimal performance in GPU applications. By understanding the different memory types, preallocating resources, batching data transfers, and leveraging coalescing techniques, developers can unlock the full potential of their applications. Every small adjustment can lead to significant performance improvements, making it vital to prioritize these aspects from the outset.

Minimizing Kernel Launch Overhead

Reducing the time spent on launching kernels is crucial for enhancing performance. Each kernel launch incurs a significant overhead that can hinder the efficiency of applications. By strategically managing these launches, a developer can achieve remarkable gains in execution time. Small kernels can be particularly problematic; their frequent launches can lead to unnecessary delays.

To effectively address this concern, consider the following approaches:

For instance, by using batch processing, developers can dramatically decrease the frequency of kernel invocations, thus allowing more computation to be completed in each launch. Instead of launching a kernel for every single operation, it's often more efficient to aggregate tasks that can be executed together, leading to a noticeable reduction in launch time.

__global__ void batchProcessKernel(int* data, int size) {

int idx = blockIdx.x * blockDim.x + threadIdx.x;

if (idx < size) {

// Perform batch operations

}

}

This type of approach can significantly improve overall throughput. By minimizing kernel launches, developers not only save time but also improve resource utilization on the device. Additionally, leveraging asynchronous operations allows for overlapping data transfers with computation, further optimizing the execution workflow.

In summary, meticulous planning of kernel launches plays a significant role in achieving superior performance. By employing techniques like batching, asynchronous execution, and kernel fusion, developers can effectively reduce overhead and improve overall application efficiency. Balancing the trade-off between launch frequency and computational workload is essential for maximizing the potential of GPU resources.

Utilizing Shared Memory Effectively

Shared memory serves as a critical resource in parallel computing environments, allowing threads within the same block to communicate quickly. This high-speed memory space enhances performance by reducing global memory access latency. When used correctly, it can lead to significant improvements in application efficiency. The key lies in understanding how to harness this capability without falling into common pitfalls. Optimizing access patterns is essential, as coalesced access can dramatically benefit performance.

Threads in a block can access shared memory much faster than global memory, which is a considerable advantage for performance-critical applications. However, to fully leverage this feature, one needs to ensure proper synchronization between threads. Race conditions can occur if multiple threads attempt to read and write to the same memory locations simultaneously. Implementing synchronization mechanisms such as __syncthreads() helps prevent such issues, ensuring data integrity and coherence.

Another aspect to consider is the size of the shared memory. Each block has a limited amount of it, thus careful planning is necessary. Allocating too much shared memory can lead to a reduction in the number of threads that can run concurrently. Therefore, balance is crucial; aim for a size that accommodates necessary data while maximizing occupancy. For example, if a kernel requires storing an intermediate result, consider keeping it in shared memory rather than global memory.

Consider the following code snippet, which demonstrates the use of shared memory to optimize a simple vector addition:

This example illustrates how shared memory holds intermediate results, significantly reducing the number of accesses to slower global memory. In this case, threads perform computations in shared memory before writing the results, which minimizes the performance impact of memory latency.

Performance gains from shared memory utilization can be substantial. Studies indicate that well-optimized applications can see speedups of 2 to 10 times when shared memory is used effectively. However, developers must be aware of the trade-offs involved. Carefully analyzing memory access patterns and thread synchronization will lead to a more responsive and efficient application.

To sum up, harnessing shared memory effectively requires not only an understanding of its capabilities but also a commitment to optimizing access patterns and synchronization. As you become more familiar with these nuances, you will likely find that the potential for performance improvement is vast.

Profiling Tools for Performance Analysis

Understanding the performance of applications is crucial in a high-performance computing environment. Proper insights into execution speed, resource utilization, and bottlenecks can lead to significant improvements. Profiling tools provide the necessary feedback to enhance performance. They help in identifying inefficient areas and optimizing resource allocation.

There are various tools available, each offering unique features and benefits. Here are some of the most popular options:

Using these tools effectively can lead to a better understanding of application performance. For instance, Nsight Compute provides metrics on memory bandwidth, instruction efficiency, and occupancy rates. These statistics are vital for pinpointing where the application may be lacking.

Additionally, many tools support both CPU and GPU profiling, enabling developers to view interactions and performance across the entire application. Utilizing a combination of these tools can yield a holistic view of performance, making it easier to prioritize optimizations and streamline development workflows.

By integrating these profiling tools into the development cycle, a developer can significantly reduce debugging time and improve overall software quality. Effective use of profiling tools can elevate performance to new heights, ensuring that applications run efficiently.

In conclusion, leveraging profiling tools is not just about improvement; it's about understanding the complete picture of performance. When teams focus on data-driven development, the results can speak for themselves. For those looking to enhance their development processes, tools and techniques are invaluable assets. Additionally, if you're considering building a top-tier development team, check out this article on how to hire software engineers who can help propel your projects forward.

Add new comment

Related articles

Related Reads on Cuda developers questions

Dive into our selected range of articles and case studies, emphasizing our dedication to fostering inclusivity within software development. Crafted by seasoned professionals, each publication explores groundbreaking approaches and innovations in creating more accessible software solutions.

Perfect for both industry veterans and those passionate about making a difference through technology, our collection provides essential insights and knowledge. Embark with us on a mission to shape a more inclusive future in the realm of software development.

Navigating CUDA Runtime Errors A Developer's Guide

Master CUDA runtime errors with our developer's survival guide, offering essential troubleshooting tips and optimization techniques for seamless performance.

Key Libraries and APIs Every CUDA and OpenGL Developer Should Know to Enhance Their Development Process

Discover essential libraries and APIs for CUDA and OpenGL developers to enhance your development workflow and streamline your graphics programming.

Effective Strategies for CUDA Development Answering Frequently Asked Questions and Sharing Expert Insights

Explore best practices for CUDA development with expert tips and answers to common queries to enhance your programming efficiency.

You will enjoy it

Recommended Articles

How to hire remote Laravel developers?

When it comes to building a successful software project, having the right team of developers is crucial. Laravel is a popular PHP framework known for its elegant syntax and powerful features. If you're looking to hire remote Laravel developers for your project, there are a few key steps you should follow to ensure you find the best talent for the job.

Software Development Services for Startups

As technology continues to advance at a rapid pace, businesses are faced with the challenge of navigating the complex tech landscape to ensure they stay competitive and innovative. One key aspect of this is choosing the right software development services to help create custom solutions that meet specific business needs.

Team Extension Services The Key to Building Scalable Development Teams

In today's fast-paced tech industry, companies are constantly under pressure to deliver cutting-edge solutions quickly and efficiently. One of the key challenges that many businesses face is finding and hiring skilled software developers to meet their development needs.

Together, we can redefine technology through accessible design and development


Parallel Code: Maximizing your Performance Potential

justin.p.mckennon

·

No matter what the purpose of your application is, one thing is certain. You want to get the most bang for your buck. You see research papers being published and presented making claims of tremendous speed increases by running algorithms on the GPU (e.g. NVIDIA Tesla), in a cluster, or on a hardware accelerator (such as the Xeon Phi or Cell BE). These architectures allow for massively parallel execution of code that, if done properly, can yield lofty performance gains.

Unlike most aspects of programming, the actual writing of the programs is (relatively) simple. Most hardware accelerators support (or are very similar to) C based programming languages. This makes hitting the ground running with parallel coding an actually doable task. While mastering the development of massively parallel code is an entirely different matter, with a basic understanding of the principles behind efficient, parallel code, one can obtain substantial performance increases compared to traditional programming and serial execution of the same algorithms.

In order to ensure that you’re getting the most bang for your buck in terms of performance increases, you need to be aware of the bottlenecks associated with coprocessor/GPU programming. Fortunately for you, I’m here to make this an easier task. By simply avoiding these programming “No-No’s” you can optimize the performance of your algorithm without having to spend hundreds of hours learning about every nook and cranny of the architecture of your choice. This series will discuss and demystify these performance-robbing bottlenecks, and provide simple ways to make these a non-factor in your application.

Parallel Thread Management – Topic #1

First and foremost, the most important thing with regard to parallel programming is the proper management of threads. Threads are the smallest sequence of programmed instructions that are able to be utilized by an operating system scheduler. Your application’s threads must be kept busy (not waiting) and non-divergent. Properly scheduling and directing threads is imperative to avoid wasting precious computing time.

Read: CUDA Parallel Thread Management, Divergence and Profiling

Host/Device Transfers and Data Movement – Topic #2

Transferring data between the host and device is a very costly move. It is not uncommon to have code making multiple transactions between the host and device without the programmer’s knowledge. Cleverly structuring code can save tons of processing time! On top of that, it is imperative to understand the cost of these host device transfers. In some cases, it may be more beneficial to run certain algorithms or pieces of code on the host due to the costly transfer time associated with farming data to the device.

Read: Profile CUDA Host-to-Device Transfers and Data Movement
Read: Optimize CUDA Host-to-Device Transfers

Cache and Shared Memory Optimizations – Topic #3

In addition to managing the threads running in your application, properly utilizing the various memory types available on your device is paramount to ensuring that you’re squeezing every drop of performance from your application. Shared memory, local memory, and register memory all have their advantages and disadvantages and need to be used very carefully to avoid wasting valuable clock cycles. Phenomena such as bank conflicts, memory spilling (too much data being placed in registers and spilling into local memory),  improper loop unrolling, as well as the amount of shared memory, all play pivotal roles in obtaining the greatest performance.

Read: GPU Memory Types and Memory Performance Comparison
Read: GPU Shared Memory Performance Optimization
Read: Avoiding GPU Memory Performance Bottlenecks

More to come…

All in all, utilizing devices like the NVIDIA GPU, Cell BE or Intel Xeon Phi to increase the performance of your application doesn’t have to be a daunting task. Over the next several posts, this blog will outline and identify effective techniques to make troubleshooting the performance leaks of your application an easy matter. Each of these common bottlenecks will be discussed in detail in an effort to provide programmers insight into how to make use of all the resources that many popular architectures provide.

You May Also Like

DGX A100 review: Throughput and Hardware Summary

When NVIDIA launched the Ampere GPU architecture, they also launched their new flagship system for HPC and deep learning – the DGX 100. This system offers exceptional performance, but also new capabilities. We’ve seen immediate interest and have already shipped to some of the first adopters. Given our early access, we wanted to share a…

Deploying GPUs for Classroom and Remote Learning

As one of NVIDIA’s Elite partners, we see a lot of GPU deployments in higher education. GPUs have been proving themselves in HPC for over a decade, and they are the de-facto standard for deep learning research. They’re also becoming essential for other types of machine learning and data science. But GPUs are not always…

What Can You Do with a $15k NVIDIA Data Science Workstation? – Change Healthcare Data Science

NVIDIA’s Data Science Workstation Platform is designed to bring the power of accelerated computing to a broad set of data science workflows. Recently, we found out what happens when you lend a talented data scientist (with a serious appetite for after-hours projects + coffee) a $15k accelerated data science tool. You can recreate a massive…


CUDA Parallel Thread Management

justin.p.mckennon

·

This post is Topic #1 in our series Parallel Code: Maximizing your Performance Potential.

Regardless of the environment or architecture you are using, one thing is certain: you must properly manage the threads running in your application to optimize performance. This post will discuss how to get the most out of your threads in a CUDA application.

CUDA Threads

CUDA threads utilize block and thread IDs to determine what data to compute. Block IDs can be 1D or 2D. Thread IDs can be 1D, 2D, or 3D. Utilizing multidimensional threads and blocks greatly simplifies memory addressing when performing operations on multidimensional data (a very common occurrence in image processing, for example). You, the programmer, declare the size of the block (between 1 and 512 concurrent threads), the number of dimensions (1D, 2D, 3D) of the block, and the block dimensions in threads. In each block, all of the threads are capable of sharing data and synchronizing. The image below depicts the CUDA grid/block/thread structure.

So, assuming you’ve got your kernel up and running, how do you properly manipulate the threads running in your application? For starters, declaring the proper block/grid size (and dimension) is paramount. The appropriate size for these parameters is hardware- and device-dependent and must be fiddled with through trial and error. There’s not really a “General Rule” for determining the values for these parameters outside of really knowing the data in your application and the limitations of your hardware. Let’s say for now, that your block and grid sizes are sufficient.

With appropriate block and grid sizes/dimensions, there are two keys to optimizing your application’s performance: thread communications and thread paths.

Thread Communications

When the threads (in the same block) in your application need to communicate or share data, there are two methods that CUDA provides: shared memory and __syncthreads().  The __syncthreads() command effectively creates a barrier in the code execution where all of the threads in a given block will wait until all of the threads in that block have reached the synchronization point. This is especially useful for ensuring that computation data is written to memory before other threads read it. Improper use of this command, however, can create deadlock conditions and cause your application to hang. Deadlocks are literally a show stopper since they will cause your application to stop dead in its tracks.

Maximizing the use of shared memory will be discussed in much greater detail in a later post. Effectively utilizing shared memory is an absolute necessity for a high-performance application. Shared memory is hundreds of times faster than global memory. A common method of scheduling computations on a device that maximizes the use of shared memory is, at a high level, relatively simple:

Structuring your code in this fashion will pay great dividends.

Thread Paths

The other aspect of managing threads is controlling the paths of your threads. In nearly every application, it is almost impossible to structure code without branches (e.g. if/else conditions). Threads in the same block that execute different pieces of code (different execution paths) as a result of branch conditions are said to be divergent. When threads within the same block have different execution paths, they must be serialized. Since all threads in a block always run the same code, if any thread executes the code inside the IF condition (or if-then-else, for loops, etc), all of the threads in that same warp (a group of 32 threads) will go through that section of code. This occurs even if they are not actually executing (when the branch condition is not met)! If half of the threads in a given warp evaluate a branch condition as true, the utilization of the execution units is only 50%, meaning that half of the threads are effectively DOING NOTHING! The actual performance impact depends on the size and frequency of these divergent branch conditions.

Divergence can be avoided when a branch condition is a function of the thread ID. An example of code that would likely produce divergence:
if(threadIdx.x > 4) { //your code }

This divergence is a result of the branch granularity being less than the warp size. By making the branch granularity a whole multiple of the warp size (instead of less than the warp size), this divergence can be completely eliminated:
if(threadIdx.x/WARP_SIZE > 4) { //your code }

Optimizing in the Real World

I know what you’re thinking: “All of this is great information, Justin, but how can I check my code for deadlocks and divergent branches?” Easy – step through every line of code in the entire application with a fine toothed comb.

Well, that doesn’t sound easy. Fortunately, the NVIDIA CUDA profiler provides a very robust means for identifying these problems in your code. There are visual and text-based versions of the profiler – I’ll be discussing the text version. From the command line, the values of four environmental variables can be set:

The CUDA profiler only supports four types of events being profiled at a time. Later posts will discuss the other event types of the profiler, but with regards to managing threads, a few event types are essential to profile:

With these set, the profiler will output the number of branches and divergent branches that are encountered when executing the application, which provides invaluable insight as to which portions of code are degrading performance. Using this information, you can tell if any of the branch conditions are causing threads to diverge. In addition to the events that were chosen to be profiled, the profiler can also output the total execution time on both the CPU and GPU for the application/kernel, which can be used to gauge performance when tweaking code. Additional functions of the CUDA profiler will be discussed throughout the next several posts.

More information on the NVIDIA CUDA profiler, including information about the visual profiler, can be found in the Profiler User’s Guide:
https://docs.nvidia.com/cuda/profiler-users-guide/index.html

You May Also Like

DGX A100 review: Throughput and Hardware Summary

When NVIDIA launched the Ampere GPU architecture, they also launched their new flagship system for HPC and deep learning – the DGX 100. This system offers exceptional performance, but also new capabilities. We’ve seen immediate interest and have already shipped to some of the first adopters. Given our early access, we wanted to share a…

Deploying GPUs for Classroom and Remote Learning

As one of NVIDIA’s Elite partners, we see a lot of GPU deployments in higher education. GPUs have been proving themselves in HPC for over a decade, and they are the de-facto standard for deep learning research. They’re also becoming essential for other types of machine learning and data science. But GPUs are not always…

What Can You Do with a $15k NVIDIA Data Science Workstation? – Change Healthcare Data Science

NVIDIA’s Data Science Workstation Platform is designed to bring the power of accelerated computing to a broad set of data science workflows. Recently, we found out what happens when you lend a talented data scientist (with a serious appetite for after-hours projects + coffee) a $15k accelerated data science tool. You can recreate a massive…


CUDA Host/Device Transfers and Data Movement

justin.p.mckennon

·

This post is Topic #2 (part 1) in our series Parallel Code: Maximizing your Performance Potential.

In post #1, I discussed a few ways to optimize the performance of your application via controlling your threads and provided some insight as to how to go about fixing some possible thread related issues in your application. In this post and the following one, I will discuss another possible major performance bottleneck: Host/Device Transfers and Data Movement.

Profiling Your CUDA Code for Timing Data

In a standard CUDA application, several steps typically occur:

In the above list, steps 2 and 4 are an absolute necessity in every CUDA application, but are also HUGE performance robbers as well. These transfers are the slowest portion of data movement involved in any aspect of GPU computing. The actual transfer speed (bandwidth) is dependent on the type of hardware you’re using, but regardless of this point, it is still the slowest. In the example code below, I will illustrate this point:

int main()
{
    const unsigned int X=1048576; //1 Megabyte
    const unsigned int bytes = X*sizeof(int);
    int *hostArray= (int*)malloc(bytes);
    int *deviceArray;
    cudaMalloc((int**)&deviceArracy,bytes);
    memset(hostArray,0,bytes);
    cudaMemcpy(deviceArray,hostArray,bytes,cudaMemcpyHostToDevice);
    cudaMemcpy(hostArray,deviceArray,bytes,cudaMemcpyDeviceToHost);

    cudaFree(deviceArray);

}

In this example, there are no operations being run on the device. The data is simply copied from the host to the device and back. I’ve named this program profilerExample.cu. To profile this code, it simply needs to be compiled with nvcc and then run with nvprof (nvprof is new in CUDA 5 – the older command line profiler can still be used in earlier versions of CUDA):

$ nvcc profilerExample.cu -o profileExample

$ nvprof ./profileExample
======== NVPROF is profiling profileExample.out...
======== Command: profileExample.o
======== Profiling result:
Time(%)     Time  Calls      Avg      Min      Max Name
  50.08 718.11us      1 718.11us 718.11us 718.11us [CUDA memcpy DtoH]
  49.92 715.94us      1 715.94us 715.94us 715.94us [CUDA memcpy HtoD]

On my desktop I run a GTX 680 graphics card. As you can see from the above results, a simple copy operation to/from the GPU takes in excess of 715 microseconds each way (a lifetime in terms of computation time). In complex applications with larger amounts of data going back and forth between the host and device many times, this can result in significant time being wasted on these transfers.

Alternative Profiling Options Using Timers

In addition to the nvprof profiler, any CPU timer can be used to measure the elapsed time of a CUDA call/function or kernel execution. It is important to note that if you’re using a CPU timer to measure the timing performance of a portion (or all) or your application, that many of the CUDA functions are asynchronous. This means that the function returns control to the associated thread prior to completing all of their work. If you’re using a CPU timer you must synchronize the CPU thread associated with the timer with the device by calling cudaDeviceSynchronize() immediately before starting and stopping the CPU timer. This blocks the CPU threads until all the CUDA calls issued by that thread have been completed. CUDA also provides its own method for timing using events. The following example code snippet illustrates how to use the CUDA event timers to profile your code:

cudaEvent_t startTime, stopTime;

float time;

cudaEventCreate(&startTime);

cudaEventCreate(&stopTime);

cudaEventRecord(startTime,0);

kernel<<<gridDimensions,numberOfThreads>>>(dataOut,dataIn,size_x,size_y,NUM_REPS);

cudaEventRecord(stopTime,0);

cudaEventSynchronize(stopTime);

cudaEventElapsedTime(&time, startTime, stopTime);

cudaEventDestroy(startTime);

cudaEventDestroy(stopTime);

In this example, the cudaEventRecord() function call places the startTime and stopTime events into the default execution stream, ‘0’. The device records a timestamp for the event when it reaches that event in the execution stream. cudaEventElapsedTime() simply returns the time in milliseconds (with roughly 0.5us resolution) between the events.

Importance of Data Transfers in CUDA Applications

Analyzing these timing results can prove to be hugely beneficial in determining which portions of your application are the most expensive in terms of time. While there are a number of factors that can make one portion of code more expensive in terms of time, a good way to increase the performance of your application is to minimize the host/device transfers.

The peak theoretical bandwidth between device memory and the device processor is significantly higher than the peak theoretical bandwidth between the host memory and device memory. Therefore, in order to get the most bang for your buck in your application, you really need to minimize these host<->device data transfers. Many programmers are unaware of the high overhead associated with these transfers and by intelligently reducing or eliminating them, you can see very large gains in performance. Try performing a ‘before and after’ type test with your code. If you have multiple transfers occurring throughout your application, try reducing this number and observe the results.

The next post in this series will identify effective ways to optimize your code and avoid numerous transfers between the host and device. Utilizing pinned/mapped memory, asynchronous transfers, and overlapping transfers with computations can yield lofty performance gains if you have many host/device transfers occurring in your application.

More information about nvprof can be located at NVIDIA’s Developer Zone: CUDA Toolkit Documentation – Profiler User’s Guide

You May Also Like

DGX A100 review: Throughput and Hardware Summary

When NVIDIA launched the Ampere GPU architecture, they also launched their new flagship system for HPC and deep learning – the DGX 100. This system offers exceptional performance, but also new capabilities. We’ve seen immediate interest and have already shipped to some of the first adopters. Given our early access, we wanted to share a…

Deploying GPUs for Classroom and Remote Learning

As one of NVIDIA’s Elite partners, we see a lot of GPU deployments in higher education. GPUs have been proving themselves in HPC for over a decade, and they are the de-facto standard for deep learning research. They’re also becoming essential for other types of machine learning and data science. But GPUs are not always…

What Can You Do with a $15k NVIDIA Data Science Workstation? – Change Healthcare Data Science

NVIDIA’s Data Science Workstation Platform is designed to bring the power of accelerated computing to a broad set of data science workflows. Recently, we found out what happens when you lend a talented data scientist (with a serious appetite for after-hours projects + coffee) a $15k accelerated data science tool. You can recreate a massive…


Optimize CUDA Host/Device Transfers

justin.p.mckennon

·

This post is Topic #2 (part 2) in our series Parallel Code: Maximizing your Performance Potential.

In my previous post, CUDA Host/Device Transfers and Data Movement, I provided an introduction into the bottlenecks associated with host/device transfers and data movement. This post will delve a bit further into the subject and provide a few nifty ways to mitigate these very costly operations.

In every single CUDA application (well any useful ones, that is) there is at the very least one host-to-device transfer and one device-to-host transfer. More complicated applications often have many transfers between the host and device. In CUDA programming, this is one of the most expensive operations in terms of timing.

So, if these host/device data transfers are so costly, how do you avoid them? Well, you can’t. But what you can do is minimize the number of transfers between host and device in your application, and mask their impact on the performance of your application.

First, any intermediate data structures that are used within your kernel should always be allocated and destroyed solely on the device. This removes the need to map these structures to host memory and removes the need to transfer this data between the host and device.

If your application has multiple host/device transfers, every effort should be made to batch these transfers into one large transfer. I like to think of this as if you were carrying groceries. Why make multiple trips out to the car when you can load up your arms and do it all at once? Most GPUs support transfer speeds between 5GB/sec and 11GB/sec.

For situations where there is no way around transferring data between host and device, more advanced techniques can be employed to lessen the impact on your application: pinned (also known as page-locked, or mapped) memory and asynchronous transfers.

Pinned Memory

The cudaHostAlloc() function allows you to allocate host memory that can be read from the device and written directly to by the device. This allocated memory is called pinned memory. Pinned memory transfers attain the highest bandwidth between the host and device. During execution, a block that requires host data only needs to wait for a small portion of the data to be transferred (when operating through pinned memory). Typical host-to-device copies make all blocks wait until all of the data associated with the copy operation is transferred. Keep in mind, however, that pinning too much memory can degrade overall system performance by reducing the amount of memory available to the system for paging operations. How much memory you can safely pin differs from system to system, so definitely experiment with this to find the optimal amount.

Asynchronous Transfers

Standard host/device transfers are known as blocking transfers. Control of the main thread is returned only after the data transfer is complete. The cudaMemcpyAsync() function is effectively a non-blocking version of the standard cudaMemcpy(). When executing an asynchronous transfer via cudaMemcpyAsync(), control is returned immediately to the main thread. If you’re not jumping up and down with excitement after hearing that, you should be!

Asynchronous transfers required pinned memory and make use of CUDA streams. In CUDA, streams are essentially sequences of operations that are performed in order on the device. Creating multiple streams is a bit more of an advanced CUDA technique, but one that must be learned if you want the most bang for your buck. With multiple streams in a single application, operations within separate streams can be overlapped, providing a great way to mask the host/device transfer time. Let’s look at an example of how using multiple streams can benefit you and your application:

cudaMemcpyAsync(deviceArray,hostArray,size,cudaMemcpyHostToDevice,0);
kernel<<>>(deviceArray);
//your code

Here, both the transfer and kernel are using the default stream, 0. During execution, the kernel will not be launched until the entire copy operation is complete and control has been returned back to the main thread. This is because both the kernel and memory copy are part of the same stream. Now, let’s look at the code using multiple streams:

cudaStreamCreate(&mystream1);
cudaStreamCreate(&mystream2);
cudaMemcpyAsync(deviceArray,hostArray,size,cudaMemcpyHostToDevice,mystream1);
kernel<<>>(otherDataArray);
//your code

By defining two new streams, we are able to make use of concurrent copy and compute. The memory copy is executing in one stream while the kernel is off in another stream, asynchronous from one another. An important note is to make sure that your device supports concurrent copy and execute before you put this in all of your code. This can be done via the deviceOverlap field of the cudaDeviceProp structure.

While this is an advanced technique, if your data can be broken into chunks and transferred in various stages, you can launch multiple kernel instances to operate on each chunk of data as it arrives on the device. Doing so will almost completely mask the transfer time between the host and device.

So, armed with the knowledge of streams, asynchronous transfers, and pinned memory, you now have some insight on how to squeeze out some more performance from your application. My next post will discuss how to efficiently make use of the available memory types accessible to you within your GPU application.

You May Also Like

DGX A100 review: Throughput and Hardware Summary

When NVIDIA launched the Ampere GPU architecture, they also launched their new flagship system for HPC and deep learning – the DGX 100. This system offers exceptional performance, but also new capabilities. We’ve seen immediate interest and have already shipped to some of the first adopters. Given our early access, we wanted to share a…

Deploying GPUs for Classroom and Remote Learning

As one of NVIDIA’s Elite partners, we see a lot of GPU deployments in higher education. GPUs have been proving themselves in HPC for over a decade, and they are the de-facto standard for deep learning research. They’re also becoming essential for other types of machine learning and data science. But GPUs are not always…

What Can You Do with a $15k NVIDIA Data Science Workstation? – Change Healthcare Data Science

NVIDIA’s Data Science Workstation Platform is designed to bring the power of accelerated computing to a broad set of data science workflows. Recently, we found out what happens when you lend a talented data scientist (with a serious appetite for after-hours projects + coffee) a $15k accelerated data science tool. You can recreate a massive…


GPU Memory Types – Performance Comparison

justin.p.mckennon

·

This post is Topic #3 (part 1) in our series Parallel Code: Maximizing your Performance Potential.

CUDA devices have several different memory spaces: Global, local, texture, constant, shared and register memory. Each type of memory on the device has its advantages and disadvantages. Incorrectly making use of the available memory in your application can can rob you of the performance you desire. With so many different types of memory, how can you be certain you’re using the correct type? Well, it is no easy task.

In terms of speed, if all the various types of device memory were to race here’s how the race would turn out:

Looking at the above list, it would seem that to have the best performance we’d only want to use register file, shared memory, and constant memory. In a simple world I’d agree with that statement. However, there are many more factors associated with choosing the best form of memory for various portions of your application.

Memory Features

The only two types of memory that actually reside on the GPU chip are register and shared memory. Local, Global, Constant, and Texture memory all reside off chip. Local, Constant, and Texture are all cached.

While it would seem that the fastest memory is the best, the other two characteristics of the memory that dictate how that type of memory should be utilized are the scope and lifetime of the memory:

How to Choose Memory Type

Knowing how and when to use each type of memory goes a long way towards optimizing the performance of your application. More often than not, it is best to make use of shared memory due to the fact that threads within the same block utilizing shared memory can communicate. Combined with its excellent performance, this makes shared memory a good ‘all around’ choice when used properly. In some cases however, it may be better to make use of the other types of available memory.

Shared Memory

A common problem arises when memory is shared: with all memory available to all threads, there will be many threads accessing the data simultaneously. To alleviate this potential bottleneck, shared memory is divided into 32 logical banks. Successive sections of memory are assigned to successive banks (see Figure 1).

Some facts about shared memory:

When there are no bank conflicts present, shared memory performance is comparable to register memory. Use it properly and shared memory will be lightning fast.

Register Memory

In most cases, accessing a register consumes zero clock cycles per instruction. However, delays can occur due to read after write dependencies and bank conflicts. The latency of read after write dependencies is roughly 24 clock cycles. For newer CUDA devices that have 32 cores per multiprocessor, it may take up to 768 threads to completely hide latency.

In addition to the read after write latency, register pressure can severely detract from the performance of the application. Register pressure occurs when there are not enough registers available for a given task. When this occurs, the data is “spilled over” using local memory. See the following posts for further details.

Local Memory

Local memory is not a physical type of memory, but an abstraction of global memory. Its scope is local to the thread and it resides off-chip, which makes it as expensive to access as global memory. Local memory is used only to hold automatic variables. The compiler makes use of local memory when it determines that there is not enough register space to hold the variable. Automatic variables that are large structures or arrays are also typically placed in local memory.

Recommendation

All in all, for most applications my recommendation is definitely to try to make use of shared memory wherever possible. It is the most versatile and easy-to-use type of memory. Shared memory allows communication between threads within a warp which can make optimizing code much easier for beginner to intermediate programmers. The other types of memory all have their place in CUDA applications, but for the general case, shared memory is the way to go.

Conclusion

So now that you know a little bit about each of the various types of memory available to you in your GPU applications, you’re ready to learn how to efficiently use them. The next post will discuss how you can optimize the use of the various types of memory throughout your application.

You May Also Like

DGX A100 review: Throughput and Hardware Summary

When NVIDIA launched the Ampere GPU architecture, they also launched their new flagship system for HPC and deep learning – the DGX 100. This system offers exceptional performance, but also new capabilities. We’ve seen immediate interest and have already shipped to some of the first adopters. Given our early access, we wanted to share a…

Deploying GPUs for Classroom and Remote Learning

As one of NVIDIA’s Elite partners, we see a lot of GPU deployments in higher education. GPUs have been proving themselves in HPC for over a decade, and they are the de-facto standard for deep learning research. They’re also becoming essential for other types of machine learning and data science. But GPUs are not always…

What Can You Do with a $15k NVIDIA Data Science Workstation? – Change Healthcare Data Science

NVIDIA’s Data Science Workstation Platform is designed to bring the power of accelerated computing to a broad set of data science workflows. Recently, we found out what happens when you lend a talented data scientist (with a serious appetite for after-hours projects + coffee) a $15k accelerated data science tool. You can recreate a massive…