GPU Optimization
Fundamentals
Cliff Woolley
Developer Technology Engineer
© NVIDIA 2013
Note: Fundamentals will apply broadly
Example performance numbers are presented for Tesla K20X,
which is based on the Kepler GK110 GPU
Same general optimization concepts apply to other GPUs, though
some parameters may be different, e.g.:
Number of SMs per GPU
Number of functional units per SM
Maximum number of concurrent warps per SM
Shared memory size per SM
Register file size per SM
Developer tools from NVIDIA help you analyze the concepts
without having to memorize parameters of each architecture

© NVIDIA 2013
GPU OPTIMIZATION FUNDAMENTALS
© NVIDIA 2013
Main Requirements for GPU Performance
Expose sufficient parallelism

Utilize parallel execution resources efficiently
Use memory system efficiently
Coalesce global memory accesses
Use shared memory where possible
Have coherent execution within warps of threads
© NVIDIA 2013
APOD: A Systematic Path to Performance
Assess
Parallelize
Optimize
Deploy
© NVIDIA 2013
Identify hotspots (total time, number of calls)
Understand scaling (strong and weak)
Assess
HOTSPOTS
© NVIDIA 2013
Parallelize Applications
Applications Libraries
Libraries Programming Languages
Programming
Languages Compiler Directives
Compiler
Directives
© NVIDIA 2013
Optimize
Profile-driven optimization

Tools:
nsight Visual Studio Edition or Eclipse Edition
nvvp
NVIDIA Visual Profiler
nvprof Command-line profiling

© NVIDIA 2013
Deploy
Check API return values
Run cuda-memcheck tools
Library distribution
Cluster management

Early gains
Subsequent changes are evolutionary
Productize

© NVIDIA 2013
ASSESS
© NVIDIA 2013
Profile the code, find the hotspot(s)
Focus your attention where it will give the most benefit
Assess
© NVIDIA 2013
Assess
We’ve found a hotspot to work on!
What percent of our total time does this represent?
How much can we improve it? What is the “speed of light”?
How much will this improve our overall performance?
© NVIDIA 2013
Assess
Let’s investigate…
Strong scaling and Amdahl’s Law
Weak scaling and Gustafson’s Law
Expected perf limiters: Bandwidth? Computation? Latency?
© NVIDIA 2013
Assess: Understanding Scaling
Strong Scaling
A measure of how, for fixed overall problem size, the time to
solution decreases as more processors are added to a system
Linear strong scaling: speedup achieved is equal to number of
processors used

Amdahl’s Law:
𝑺=
𝟏
𝟏−𝑷+ 𝑷
𝑵
 ≈
𝟏
(𝟏−𝑷)
© NVIDIA 2013
Assess: Understanding Scaling
Weak Scaling
A measure of how time to solution changes as more processors
are added with fixed problem size per processor
Linear weak scaling: overall problem size increases as num. of
processors increases, but execution time remains constant

Gustafson’s Law:

𝑺= 𝑵+ (𝟏−𝑷)(𝟏−𝑵)
© NVIDIA 2013
Assess: Applying Strong and Weak Scaling
Understanding which type of scaling is most applicable is an
important part of estimating speedup:
Sometimes problem size will remain constant
Other times problem size will grow to fill the available processors

Apply either Amdahl's or Gustafson's Law to determine an upper
bound for the speedup
© NVIDIA 2013
Assess: Applying Strong Scaling
Recall that in this case we are wanting to optimize an
existing kernel with a pre-determined workload

That’s strong scaling, so Amdahl’s Law will determine
the maximum speedup

~93%
© NVIDIA 2013
Assess: Applying Strong Scaling
Say, for example, our kernel is ~93% of total time:
Speedup 𝑺=
𝟏
𝟏−𝑷+ 𝑷
𝑺𝑷
     (SP = speedup in parallel part)
In the limit when 𝑺𝑷 is huge, 𝑺 will approach
𝟏
𝟏−𝟎.𝟗𝟑≈𝟏𝟒. 𝟑
In practice, it will be less than that depending on the 𝑺𝑷 achieved
Getting 𝑺𝑷 to be high is the goal of optimizing, of course
~93%
© NVIDIA 2013
Assess: Speed of Light
What’s the limiting factor?
Memory bandwidth?
Compute throughput?
Latency?

Not sure?
Get a rough estimate by counting bytes per instruction,
compare it to “balanced” peak ratio
𝑮𝑩𝒚𝒕𝒆𝒔/𝒔𝒆𝒄
 𝑮𝒊𝒏𝒔𝒏𝒔/𝒔𝒆𝒄
Profiler will help you determine this

© NVIDIA 2013
Assess: Limiting Factor
Comparing bytes per instr. will give you a guess as to whether
you’re likely to be bandwidth-bound or instruction-bound

Comparing actual achieved GB/s vs. theory and achieved
Ginstr/s vs. theory will give you an idea of how well you’re doing
If both are low, then you’re probably latency-bound and need to expose
more (concurrent) parallelism
© NVIDIA 2013
Assess: Limiting Factor
© NVIDIA 2013
Assess: Speed of Light
What’s the limiting factor?
Memory bandwidth? Compute throughput? Latency?

Consider SpMV: intuitively expect it to be bandwidth-limited
Say we discover we’re getting only ~38% of peak bandwidth
If we aim to get this up to ~65% of peak, that’s 1.7 for this kernel
1.7  for this kernel translates into 1.6 overall due to Amdahl:

𝐒=
𝟏
𝟏−𝟎.𝟗𝟑+𝟎.𝟗𝟑
𝟏.𝟕
≈𝟏. 𝟔
~93%
© NVIDIA 2013
Assess: Limiting Factor
For our example SpMV kernel, our first discovery was that we’re
latency-limited, not bandwidth, since utilization was so low

This tells us our first “optimization” step actually needs to be
related how we expose (memory-level) parallelism
~93%
© NVIDIA 2013
PARALLELIZE
© NVIDIA 2013
PARALLELIZE
Computation
© NVIDIA 2013
Parallelize Applications
Applications Libraries
Libraries Programming Languages
Programming
Languages Compiler Directives
Compiler
Directives
Pick the best tool for the job
© NVIDIA 2013
NVIDIA cuFFT
NVIDIA cuSPARSE
NVIDIA cuBLAS
NVIDIA cuRAND
NVIDIA NPP
Vector Signal
Image Processing
Matrix Algebra on
GPU and Multicore
C++ Templated
Parallel Algorithms
IMSL Library
GPU Accelerated
Linear Algebra
Building-block
Algorithms
CenterSpace NMath Parallelize: e.g., with GPU Accelerated Libraries
Parallelize: e.g., with GPU Accelerated Libraries
© NVIDIA 2013

// generate 32M random numbers on host
thrust::host_vector<int> h_vec(32 << 20);
thrust::generate(h_vec.begin(),
                 h_vec.end(),
                 rand);

// transfer data to device (GPU)
thrust::device_vector<int> d_vec = h_vec;

// sort data on device
thrust::sort(d_vec.begin(), d_vec.end());

// transfer data back to host
thrust::copy(d_vec.begin(),
             d_vec.end(),
             h_vec.begin());
 Parallelize: e.g., with Thrust
Parallelize: e.g., with Thrust
Similar to C++ STL
High-level interface
Enhances developer productivity
Enables performance portability
between GPUs and multicore CPUs
Flexible
Backends for CUDA, OpenMP, TBB
Extensible and customizable
Integrates with existing software
Open source
thrust.github.com or developer.nvidia.com/thrust
© NVIDIA 2013
Parallelize: e.g., with OpenACC
Program myscience
   ... serial code ...
!$acc kernels
   do k = 1,n1
      do i = 1,n2
          ... parallel code ...
      enddo
    enddo
!$acc end kernels
  ...
End Program myscience
CPU
GPU
Your original
Fortran or C code Directives-based approach Compiler parallelizes code Works on many-core GPUs & multicore CPUs
Directives-based approach
Compiler parallelizes code
Works on many-core GPUs &
multicore CPUs OpenACC
OpenACCCompiler
Compiler  Directive
Directive
www.nvidia.com/gpudirectives
© NVIDIA 2013

void saxpy_serial(int n,
                  float a,
                  float *x,
                  float *y)
{

  for (int i = 0; i < n; ++i)
    y[i] = a*x[i] + y[i];
}

// Perform SAXPY on 1M elements
saxpy_serial(4096*256, 2.0, x, y);
__global__
void saxpy_parallel(int n,
                    float a,
                    float *x,
                    float *y)
{
  int i = blockIdx.x * blockDim.x +
          threadIdx.x;
  if (i < n) y[i] = a*x[i] + y[i];
}

// Perform SAXPY on 1M elements
saxpy_parallel<<<4096,256>>>(n,2.0,x,y); Parallelize: e.g., with CUDA C
Parallelize: e.g., with CUDA C
Standard C Code
CUDA C Code
developer.nvidia.com/cuda-toolkit
© NVIDIA 2013
Parallelism Needed
GPU is a parallel machine
Lots of arithmetic pipelines
Multiple memory banks

To get good performance, your code must expose sufficient
parallelism for 2 reasons:
To actually give work to all the pipelines
To hide latency of the pipelines

Rough rule of thumb for Tesla K20X:
You want to have 14K or more threads running concurrently
© NVIDIA 2013
void transpose(float in[][], float out[][], int N)
{
  for(int j=0; j < N; j++)
    for(int i=0; i < N; i++)
      out[j][i] = in[i][j];
}
Case Study: Matrix Transpose
i
j
© NVIDIA 2013
+ Quickly implemented                                      - Performance weak
Need to expose parallelism!
An Initial CUDA Version
__global__ void transpose(float in[], float out[], int N)
{
  for(int j=0; j < N; j++)
     for(int i=0; i < N; i++)
       out[i*N+j] = in[j*N+i];

}

float in[N*N], out[N*N];
…
transpose<<<1,1>>>(in, out, N);
© NVIDIA 2013
+ Quickly implemented                                      - Performance weak
Need to expose parallelism!
An Initial CUDA Version
__global__ void transpose(float in[], float out[], int N)
{
  for(int j=0; j < N; j++)
     for(int i=0; i < N; i++)
       out[i*N+j] = in[j*N+i];

}

float in[N*N], out[N*N];
…
transpose<<<1,1>>>(in, out, N);
© NVIDIA 2013
Parallelize across matrix elements
tid
in
tid
tid
tid
 out
tid
tid
__global__ transpose(float in[], float out[])
{
  int tid = threadIdx.x;
  int bid = blockIdx.x;

  out[tid*N+bid] = in[bid*N+tid];
}

float in[], out[];
…
transpose<<<N,N>>>(in, out);
Process elements independently
bid
bid
bid bid
© NVIDIA 2013
PARALLELIZE
Data Transfer
© NVIDIA 2013
Heterogeneous system: overlap work and data movement
Asynchronicity = Overlap = Parallelism
DMA
DMA
© NVIDIA 2013
This is the kind of case we would be concerned about
Found the top kernel, but the GPU is mostly idle – that is our bottleneck
Need to overlap CPU/GPU computation and PCIe transfers
Asynchronicity
© NVIDIA 2013
What we want to see is maximum overlap of all engines
Parallelize: Achieve Asynchronicity
© NVIDIA 2013
OPTIMIZE
© NVIDIA 2013
Main Requirements for GPU Performance
Expose sufficient parallelism

Utilize parallel execution resources efficiently
Use memory system efficiently
Coalesce global memory accesses
Use shared memory where possible
Have coherent execution within warps of threads
© NVIDIA 2013
GPU Optimization Fundamentals
Find ways to parallelize sequential code
Adjust kernel launch configuration to maximize device utilization
Ensure global memory accesses are coalesced
Minimize redundant accesses to global memory
Avoid different execution paths within the same warp
Minimize data transfers between the host and the device

http://docs.nvidia.com/cuda/cuda-c-best-practices-guide/
© NVIDIA 2013
GPU Optimization Fundamentals
Find ways to parallelize sequential code

Kernel optimizations
Launch configuration
Global memory throughput
Shared memory access
Instruction throughput / control flow

Optimization of CPU-GPU interaction
Maximizing PCIe throughput
Overlapping kernel execution with memory copies
© NVIDIA 2013
OPTIMIZE
Kernel Optimizations: Kernel Launch Configuration
© NVIDIA 2013
Kernel Launch Configuration
A kernel is a function that runs on the GPU
A kernel is launched as a grid of blocks of threads
Launch configuration is the number of blocks and number of
threads per block, expressed in CUDA with the <<< >>> notation:

mykernel<<<blocks_per_grid,threads_per_block>>>(…);

What values should we pick for these?
Need enough total threads to process entire input
Need enough threads to keep the GPU busy
Selection of block size is an optimization step involving warp occupancy
© NVIDIA 2013
High-level view of GPU Architecture
Several Streaming Multiprocessors
E.g., Kepler GK110 has up to 15 SMs
L2 Cache shared among SMs
Multiple channels to DRAM
Kepler GK110
© NVIDIA 2013
Kepler Streaming Multiprocessor (SMX)
Per SMX:
192 SP CUDA Cores
64 DP CUDA Cores
4 warp schedulers
Up to 2048 concurrent threads
One or two instructions issued
per scheduler per clock from a
single warp
Register file (256KB)
Shared memory (48KB)
© NVIDIA 2013
CUDA Execution Model
Thread: Sequential execution unit
All threads execute same sequential program
Threads execute in parallel

Threads Block: a group of threads
Executes on a single Streaming Multiprocessor (SM)
Threads within a block can cooperate
Light-weight synchronization
Data exchange

Grid: a collection of thread blocks
Thread blocks of a grid execute across multiple SMs
Thread blocks do not synchronize with each other
Communication between blocks is expensive
© NVIDIA 2013
Software
Hardware
Threads are executed by scalar CUDA Cores
Thread
CUDA
Core
Thread Block
Multiprocessor
Thread blocks are executed on multiprocessors

Thread blocks do not migrate

Several concurrent thread blocks can reside on
one multiprocessor - limited by multiprocessor
resources (shared memory and register file)
Grid
A kernel is launched as a grid of thread blocks
Execution Model
Device
© NVIDIA 2013
Launch Configuration: General Guidelines
How many blocks should we use?
1,000 or more thread blocks is best
Rule of thumb: enough blocks to fill the GPU at least 10s of times over
Makes your code ready for several generations of future GPUs

© NVIDIA 2013
Launch Configuration: General Guidelines
How many threads per block should we choose?
The really short answer: 128, 256, or 512 are often good choices

The slightly longer answer:
Pick a size that suits the problem well
Multiples of 32 threads are best
Pick a number of threads per block (and a number of blocks) that is
sufficient to keep the SM busy

© NVIDIA 2013
Multiprocessor
32 Threads
Warps
A thread block consists
of warps of 32 threads

A warp is executed
physically in parallel on
some multiprocessor.

Threads of a warp issue
instructions in lock-
step (as with SIMD)
=
Warps
Thread Block
32 Threads
32 Threads
32 Threads
© NVIDIA 2013
Hardware Levels of Parallelism
SIMD
MPI
Single Instruction, Multiple Data
In-core parallelism
SMT
Simultaneous Multithreading
Cross-core, Cross-socket
Single Computer
OpenMP, pthreads

Multiple “computers”
Tightly-coupled
Supercomputing apps

SIMT
Single Instruction, Multiple Threads
In-processor parallelism
Many threads on many cores
These form a continuum. Best performance is achieved with a mix.
© NVIDIA 2013
Low Latency or High Throughput?
CPU
Optimized for low-latency
access to cached data sets
Control logic for out-of-order
and speculative execution

GPU
Optimized for data-parallel,
throughput computation
Architecture tolerant of
memory latency
More transistors dedicated to
computation

© NVIDIA 2013
Occupancy
Need enough concurrent warps
per SM to hide latencies:
Instruction latencies
Memory access latencies

Hardware resources determine
number of warps that fit per SM

Occupancy = Nactual / Nmax

© NVIDIA 2013
Low Latency or High Throughput?
CPU architecture must minimize latency within each thread
GPU architecture hides latency with computation from other (warps of) threads
GPU Streaming Multiprocessor – High-throughput Processor
CPU core – Low-latency Processor
Computation Thread/Warp
Tn

Processing
Waiting for data
Ready to be processed
Context switch
W1

W2

W3

W4

T1

T2

T3

T4

© NVIDIA 2013
Latency Hiding
Instruction latencies:
Roughly 10-20 cycles for arithmetic operations
DRAM accesses have higher latencies (400-800 cycles)
Instruction Level Parallelism (ILP)
Independent instructions between two dependent ones
ILP depends on the code, done by the compiler
Switching to a different warp
If a warp must stall for N cycles due to dependencies, having N other
warps with eligible instructions keeps the SM going
Switching among concurrently resident warps has no overhead
State (registers, shared memory) is partitioned, not stored/restored
     FFMA R0, R43, R0, R4;
     FFMA R1, R43, R4, R5;
     FMUL R7, R9, R0;
     FMUL R8, R9, R1;
     ST.E [R2], R7;
ILP=2
© NVIDIA 2013
Occupancy
Occupancy: number of concurrent warps per SM, expressed as:
Absolute number of warps of threads that fit concurrently (e.g., 1..64), or
Ratio of warps that fit concurrently to architectural maximum (0..100%)

Number of warps that fit determined by resource availability:
Threads per thread block
Registers per thread
Shared memory per thread block
Kepler SM resources:

–
64K 32-bit registers
–
Up to 48 KB of shared memory
–
Up to 2048 concurrent threads
–
Up to 16 concurrent thread blocks
© NVIDIA 2013
Occupancy and Performance
Note that 100% occupancy isn’t needed to reach maximum
performance
Once the “needed” occupancy (enough warps to switch among to cover
latencies) is reached, further increases won’t improve performance

Level of occupancy needed depends on the code
More independent work per thread -> less occupancy is needed
Memory-bound codes tend to need more occupancy
Higher latency than for arithmetic, need more work to hide it
© NVIDIA 2013
Thread Block Size and Occupancy
Thread block size is a multiple of warp size (32)
Even if you request fewer threads, hardware rounds up
Thread blocks can be too small
Kepler SM can run up to 16 thread blocks concurrently
SM can reach the block count limit before reaching good occupancy
E.g.: 1-warp blocks = 16 warps/SM on Kepler (25% occ – probably not enough)
Thread blocks can be too big
Enough SM resources for more threads, but not enough for a whole block
A thread block isn’t started until resources are available for all of its threads

© NVIDIA 2013
Thread Block Sizing
SM resources:
Registers
Shared memory
Number of warps allowed by SM resources
Too few
threads per block
Too many
threads per block
© NVIDIA 2013
CUDA Occupancy Calculator

Analyze effect of
resource consumption
on occupancy

© NVIDIA 2013
Occupancy Analysis in NVIDIA Visual Profiler

Occupancy here is limited
by grid size and number of
threads per block

© NVIDIA 2013
OPTIMIZE
Kernel Optimizations: Global Memory Throughput
© NVIDIA 2013
Host
CPU
Chipset
DRAM
Device
DRAM

Global
Constant
Texture
Local
GPU
Multiprocessor
Registers
Shared Memory
Multiprocessor
Registers
Shared Memory
Multiprocessor
Registers
Shared Memory
Constant and Texture
Caches
L1 / L2 Cache
CUDA Memory Architecture
© NVIDIA 2013
Optimizing Memory Throughput
Goal: utilize all available memory
bandwidth

Little’s Law:
# bytes in flight = latency * bandwidth

Increase parallelism (bytes in flight)

(or)
Reduce latency (time between requests)
Access latency L
© NVIDIA 2013
Illustration: Little’s Law for Escalators
Say the parameters of our escalator are:
1 person fits on each step
Step arrives every 2 secs (bandwidth=0.5 persons/s)
20 steps tall (latency=40 seconds)
1 person in flight: 0.025 persons/s achieved
To saturate bandwidth:
Need 1 person arriving every 2 s
Means we’ll need 20 persons in flight
The idea: Bandwidth × Latency
It takes latency time units for the first person to arrive
We need bandwidth persons to get on the escalator every time unit
© NVIDIA 2013
Memory-Level Parallelism = Bandwidth
In order to saturate memory bandwidth, SM must have
enough independent memory requests in flight concurrently
© NVIDIA 2013
Memory-Level Parallelism: Requests in flight
Achieved Kepler memory throughput
Shown as a function of number of concurrent requests
per SM with 128-byte lines
© NVIDIA 2013
Experiment: vary size of accesses by
threads of a warp, check performance
Memcopy kernel: each warp has 2 concurrent
requests (one write and the read following it)
Accesses by a warp:

4B words: 1 line

8B words: 2 lines
 16B words: 4 lines

To achieve same
throughput at lower
occupancy or with
smaller words, need
more independent
requests per warp
Requests per Thread and Performance
© NVIDIA 2013
Optimizing Access Concurrency
Ways to increase concurrent accesses:
Increase occupancy (run more warps concurrently)
Adjust block dimensions to maximize occupancy
If occupancy is limited by registers per thread, try to reduce register count
(-maxrregcount option or __launch_bounds__)

Modify code to process several elements per thread
Doubling elements per thread doubles independent accesses per thread
© NVIDIA 2013
OPTIMIZE
Kernel Optimizations: Global Memory Access Coalescing
© NVIDIA 2013
Mechanics of a Memory Access
Memory operations are issued per warp
Just like all other instructions

Operation:
Threads in a warp provide memory addresses
Hardware determines which lines/segments are needed, fetches them
© NVIDIA 2013
Memory Access Efficiency Analysis
Two perspectives on the throughput:
Application’s point of view: count only bytes requested by application
HW point of view: count all bytes moved by hardware

The two views can be different:
Memory is accessed at 32 byte granularity
With a scattered or offset pattern, the application doesn’t use all the bytes the
hardware actually transferred
Broadcast: the same small transaction serves many threads in a warp
© NVIDIA 2013
Access Patterns vs. Memory Throughput
Scenario:
Warp requests 32 aligned, consecutive 4-byte words
Addresses fall within 4 segments
Warp needs 128 bytes
128 bytes move across the bus
Bus utilization: 100%
...
addresses from a warp
96
192
128
160
224
288
256
32
64
352
320
384
448
416
Memory addresses
0
© NVIDIA 2013
Access Patterns vs. Memory Throughput
...
addresses from a warp
Scenario:
Warp requests 32 aligned, permuted 4-byte words
Addresses fall within 4 segments
Warp needs 128 bytes
128 bytes move across the bus
Bus utilization: 100%

96
192
128
160
224
288
256
32
64
352
320
384
448
416
Memory addresses
0
© NVIDIA 2013
Access Patterns vs. Memory Throughput
Scenario:
Warp requests 32 misaligned, consecutive 4-byte words
Addresses fall within at most 5 segments
Warp needs 128 bytes
At most 160 bytes move across the bus
Bus utilization: at least 80%
Some misaligned patterns will fall within 4 segments, so 100% utilization
96
192
128
160
224
288
256
32
64
352
320
384
448
416
Memory addresses
0
...
addresses from a warp
© NVIDIA 2013
Access Patterns vs. Memory Throughput
addresses from a warp
Scenario:
All threads in a warp request the same 4-byte word
Addresses fall within a single segment
Warp needs 4 bytes
32 bytes move across the bus
Bus utilization: 12.5%

...
96
192
128
160
224
288
256
32
64
352
320
384
448
416
Memory addresses
0
© NVIDIA 2013
Access Patterns vs. Memory Throughput
addresses from a warp
96
192
128
160
224
288
256
32
64
352
320
384
448
416
Memory addresses
0
Scenario:
Warp requests 32 scattered 4-byte words
Addresses fall within N segments
Warp needs 128 bytes
N*32 bytes move across the bus
Bus utilization:  128 / (N*32)
...
© NVIDIA 2013
Parallelizing SAXPY
void saxpy(int n, float a, float * x, float
* y)
{
  for(int i=0; i<n; i++)
  {

y[base +i] += a * x[base+ i];
  }
}

Divide the work equally
among T threads
Each thread is responsible for
computing one contiguous
‘region’ of the arrays
This is good for pthreads
© NVIDIA 2013
Parallelizing SAXPY
__global__ void saxpy1(int n, float a, float
* x, float * y)
{
  int workPerThread = 1 + n/blockDim.x;
  int base = threadIdx.x * workPerThread;

  for(int i=0; i<workPerThread; i++)
  {
    if(base + i < n)
    {
      y[base +i] += a * x[base+ i];
    }
  }
}

Divide the work equally
among T threads
Each thread is responsible for
computing one contiguous
‘region’ of the arrays
This is good for pthreads

thread 0
thread 1
thread 2
thread 3
…
thread 31
x
© NVIDIA 2013
Parallelizing SAXPY
__global__ void saxpy1(int n, float a, float
* x, float * y)
{
  int workPerThread = 1 + n/blockDim.x;
  int base = threadIdx.x * workPerThread;

  for(int i=0; i<workPerThread; i++)
  {
    if(base + i < n)
    {
      y[base +i] += a * x[base+i];
    }
  }
}

In SIMT, 32 threads of a warp
issue the x[base+i] instruction
simultaneously.
Each thread has different value
of base
if workPerThread > 1, this
becomes a strided load
thread 0
thread 1
thread 2
thread 3
…
thread 31
x
© NVIDIA 2013
Parallelizing SAXPY
__global__ void saxpy1(int n, float a, float
* x, float * y)
{
  int workPerThread = 1 + n/blockDim.x;
  int base = threadIdx.x * workPerThread;

  for(int i=0; i<workPerThread; i++)
  {
    if(base + i < n)
    {
      y[base +i] += a * x[base+i];
    }
  }
}

In SIMT, 32 threads of a warp
issue the x[base+i] instruction
simultaneously.
Each thread has different value
of base
if workPerThread > 1, this
becomes a strided load
thread 0
thread 1
thread 2
thread 3
…
thread 31
x
© NVIDIA 2013
…
A Better Way to Parallelize SAXPY
Divide work up so that each
pass through the loop, the
thread block computes one
‘contiguous region’ of the
array.
Achieves memory coalescing

loopcount = 0
loopcount = 1
…
x
loopcount=k
__global__ void saxpy2(int n, float a, float
* x, float * y)
{
  int id;
  int loopCount = 0;
  while(id < n)
  {
    id = loopCount*blockDim.x + threadIdx.x;
    y[id] += a * x[id];
    loopCount++;
  }
}

© NVIDIA 2013
__global__ void saxpy2(int n, float a, float
* x, float * y)
{
  int id;
  int loopCount = 0;
  while(id < n)
  {
    id = loopCount*blockDim.x + threadIdx.x;
    y[id] += a * x[id];
    loopCount++;
  }
}

The area of X addressed by
each warp is contiguous in
global memory.
The number of global memory
transactions is minimized.
This effect translates to loads
and stores of y also.
loopcount = 0
loopcount = 1
…
…
x
A Better Way to Parallelize SAXPY
loopcount=k
© NVIDIA 2013
Structures of Non-Native Size
Say we are reading a 12-byte structure per thread

struct Position
{

float x, y, z;
};
...
__global__ void kernel( Position *data, ... )
{

int idx = blockIdx.x * blockDim.x + threadIdx.x;

Position temp = data[idx];

...
}
© NVIDIA 2013
Structure of Non-Native Size
Compiler converts temp = data[idx] into 3 loads:
Each loads 4 bytes
Can’t do an 8 and a 4 byte load: 12 bytes per element means that every
other element wouldn’t align the 8-byte load on 8-byte boundary
Addresses per warp for each of the loads:
Successive threads read 4 bytes at 12-byte stride
© NVIDIA 2013
First Load Instruction
4
8
12 16 20
56 60 64
0
24
48 52
36 40 44
28 32
addresses from a warp
...
© NVIDIA 2013
Second Load Instruction
4
8
12 16 20
56 60 64
0
24
48 52
36 40 44
28 32
addresses from a warp
...
© NVIDIA 2013
Third Load Instruction
4
8
12 16 20
56 60 64
0
24
48 52
36 40 44
28 32
addresses from a warp
...
© NVIDIA 2013
Performance and Solutions
Because of the address pattern, we end up moving 3x more bytes
than application requests
We waste a lot of bandwidth, leaving performance on the table
Potential solutions:
Change data layout from array of structures to structure of arrays
In this case: 3 separate arrays of floats
The most reliable approach (also ideal for both CPUs and GPUs)
Use loads via read-only cache
As long as lines survive in the cache, performance will be nearly optimal
Stage loads via shared memory
© NVIDIA 2013
Global Memory Access Patterns
SoA vs AoS:
Good:
point.x[i]
Not so good: point[i].x

Strided array access:
~OK:
x[i] = a[i+1]  – a[i]
Slower:
x[i] = a[64*i] – a[i]

Random array access:
Slower:
a[rand(i)]
0
1
31
0
1
31
© NVIDIA 2013
Summary: GMEM Optimization
Strive for perfect address coalescing per warp
Align starting address (may require padding)
A warp will ideally access within a contiguous region
Avoid scattered address patterns or patterns with large strides between
threads
Analyze and optimize address patterns:
Use profiling tools (included with CUDA toolkit download)
Compare the transactions per request to the ideal ratio
Choose appropriate data layout (prefer SoA)
If needed, try read-only loads, staging accesses via SMEM
© NVIDIA 2013
A note about caches
L1 and L2 caches
Ignore in software design
Thousands of concurrent
threads – cache blocking
difficult at best

Read-only Data Cache
Shared with texture pipeline
Useful for uncoalesced reads
Handled by compiler when
const __restrict__ is used, or
use __ldg() primitive
© NVIDIA 2013
Blocking for GPU Memory Caches
Short answer: DON’T
GPU caches are not intended for the same use as CPU caches
Smaller size (especially per thread), so not aimed at temporal reuse
Intended to smooth out some access patterns, help with spilled registers,
etc.
Usually not worth trying to cache-block like you would on CPU
100s to 1,000s of run-time scheduled threads competing for the cache
If it is possible to block for L1 then it’s possible block for SMEM
Same size
Same or higher bandwidth
Guaranteed locality: hw will not evict behind your back
© NVIDIA 2013
Read-only Data Cache
Go through the read-only cache
Not coherent with writes
Thus, addresses must not be written by the same kernel
Two ways to enable:
Decorating pointer arguments as hints to compiler:
Pointer of interest: const __restrict__
All other pointer arguments: __restrict__
– Conveys to compiler that no aliasing will occur
Using __ldg() intrinsic
Requires no pointer decoration
© NVIDIA 2013
Read-only Data Cache
Go through the read-only cache
Not coherent with writes
Thus, addresses must not be written by the same kernel
Two ways to enable:
Decorating pointer arguments as hints to compiler:
Pointer of interest: const __restrict__
All other pointer arguments: __restrict__
– Conveys to compiler that no aliasing will occur
Using __ldg() intrinsic
Requires no pointer decoration

__global__ void kernel(
              int* __restrict__ output,
        const int* __restrict__ input )
{
     ...
     output[idx] = input[idx];
}
© NVIDIA 2013
Read-only Data Cache
Go through the read-only cache
Not coherent with writes
Thus, addresses must not be written by the same kernel
Two ways to enable:
Decorating pointer arguments as hints to compiler:
Pointer of interest: const __restrict__
All other pointer arguments: __restrict__
– Conveys to compiler that no aliasing will occur
Using __ldg() intrinsic
Requires no pointer decoration

__global__ void kernel( int *output,
                        int *input )
{
     ...
     output[idx] = __ldg( &input[idx] );
}
© NVIDIA 2013
Texture and Constant Memory
Read-only
Data resides in global memory
Read via special-purpose caches
© NVIDIA 2013
Texture
Separate cache
Dedicated texture cache hardware provides:
Out-of-bounds index handling
clamp or wrap-around
Optional interpolation
Think: using fp indices for arrays
Linear, bilinear, trilinear
– Interpolation weights are 9-bit
Optional format conversion
{char, short, int} -> float
All of these are “free”
© NVIDIA 2013
Examples of Texture Object Indexing
©
11
Index Clamp:
0    1    2    3    4
1
2
3
0
(5.5, 1.5)
1
2
3
0
(2.5, 0.5)
(1.0, 1.0)
0    1    2    3    4
1
2
3
0
(5.5, 1.5)
0    1    2    3    4
Index Wrap:
Integer indices fall between
elements
Optional interpolation:
    Weights are determined by coordinate distance
© NVIDIA 2013
OPTIMIZE
Kernel Optimizations: Shared Memory Accesses
© NVIDIA 2013
Shared Memory
Fast, on-chip memory
Accessible by all threads within a thread block
Common allocation for entire thread block
Variety of uses:
Software managed cache (e.g., tiled DGEMM)
Global memory coalescing (e.g., transpose)
Communication within a thread block  (e.g., FFT, reductions)
Limited Resource
Use of shared memory affects occupancy
Registers
L1 SM
SM
SMEM
© NVIDIA 2013
Shared Memory Organization
Organized in 32 independent banks

Optimal access: no two words from
same bank
Separate banks per thread
Banks can multicast

Multiple words from same bank serialize

C
Bank
Any 1:1 or multicast pattern
C
C
C
Bank
Bank
Bank
C
Bank
C
C
C
Bank
Bank
Bank
© NVIDIA 2013
Bank Addressing Examples

No Bank Conflicts

No Bank Conflicts
Bank 31
Bank 7
Bank 6
Bank 5
Bank 4
Bank 3
Bank 2
Bank 1
Bank 0
Thread 31
Thread 7
Thread 6
Thread 5
Thread 4
Thread 3
Thread 2
Thread 1
Thread 0
Bank 31
Bank 7
Bank 6
Bank 5
Bank 4
Bank 3
Bank 2
Bank 1
Bank 0
Thread 31
Thread 7
Thread 6
Thread 5
Thread 4
Thread 3
Thread 2
Thread 1
Thread 0
© NVIDIA 2013
Bank Addressing Examples

2-way Bank Conflicts

8-way Bank Conflicts
Thread 31
Thread 30
Thread 29
Thread 28
Thread 4
Thread 3
Thread 2
Thread 1
Thread 0
Bank 31
Bank 7
Bank 6
Bank 5
Bank 4
Bank 3
Bank 2
Bank 1
Bank 0
Thread 31
Thread 7
Thread 6
Thread 5
Thread 4
Thread 3
Thread 2
Thread 1
Thread 0
Bank 9
Bank 8
Bank 31
Bank 7
Bank 2
Bank 1
Bank 0
x8
x8
© NVIDIA 2013
Motivating Example: Matrix Transpose
_global__ void gpuTranspose_kernel(int rows,
int cols, float *in, float *out)
{
  int i, j;
  i = blockIdx.x * blockDim.x + threadIdx.x;
  j = blockIdx.y * blockDim.y + threadIdx.y;
  out[i * rows + j] = in[j * cols + i];
}

Either write or read is strided in gmem
and uncoalesced
Solution: tile in shared memory

i
j
© NVIDIA 2013
Transposing with Shared Memory
1. Read block_ij into
shared memory
•
Reads are coalesced
2. Transpose shared
memory indices
3. Write transposed
block to global
memory
•
Writes are coalesced
i
j
Global
Memory
Shared
Memory
© NVIDIA 2013
Shared Memory Organization
Organized in 32 independent banks
Note: same as warp size. Not a coincidence.
Every 32byte word is in the next bank,
modulo 32.
Optimal access: no two words from
same bank
Separate banks per thread
Banks can multicast

Multiple words from same bank serialize
Called bank conflict, causes instruction replay
C
Bank
Any 1:1 or multicast pattern
C
C
C
Bank
Bank
Bank
C
Bank
C
C
C
Bank
Bank
Bank
© NVIDIA 2013
Shared Memory: Avoiding Bank Conflicts
Example: 32x32 SMEM array
Warp accesses a column:
32-way bank conflicts (threads in a warp access the same bank)

31
2
1
0
31
2
1
0
31
2
1
0
Bank 0
Bank 1
  …
Bank 31
2
0
1
31
© NVIDIA 2013
Shared Memory: Avoiding Bank Conflicts
Example: 32x32 SMEM array
Warp accesses a column:
32-way bank conflicts (threads in a warp access the same bank)

31
2
1
0
31
2
1
0
31
2
1
0
Bank 0
Bank 1
  …
Bank 31
2
0
1
31
Accesses along row
produces 0 bank
conflicts
Accesses along
column produces 32
bank conflicts
(replays)
© NVIDIA 2013
Shared Memory: Avoiding Bank Conflicts
Add a column for padding:
32x33 SMEM array
Warp accesses a column:
32 different banks, no bank conflicts

31
2
1
0
31
2
1
0
31
2
1
0
padding
Bank 0
Bank 1
  …
Bank 31
31
2
0
1
Accesses along row
produces no bank
conflicts
Accesses along
column produces no
bank conflicts
© NVIDIA 2013
Shared Memory/L1 Sizing
Shared memory and L1 use the same 64KB physical memory
Program-configurable split:
Fermi:  48:16, 16:48
Kepler: 48:16, 16:48, 32:32
CUDA API: cudaDeviceSetCacheConfig(), cudaFuncSetCacheConfig()
Large L1 can improve performance when:
Spilling registers (more lines in the cache -> fewer evictions)
Large SMEM can improve performance when:
Occupancy is limited by SMEM
© NVIDIA 2013
Final Notes on Shared Memory
Fast: high bandwidth, low latency
Useful as user managed cache for coalescing, caching, and
communication within a thread block
Shared memory size / L1 cache size is API-configurable
16k L1 / 48k Shared (default on both Fermi and Kepler)
48k L1 / 16k Shared
32k L1 / 32k Shared (Kepler only).
Be careful of:
Overuse:  Excessive allocation can hurt occupancy
Access pattern: Lots of bank conflicts can hurt performance
© NVIDIA 2013
OPTIMIZE
Kernel Optimizations: Instruction Throughput / Control Flow
© NVIDIA 2013
Exposing Sufficient Parallelism
What SMX ultimately needs:
Sufficient number of independent instructions
Kepler GK110 is “wider” than Fermi or GK104; needs more parallelism

Two ways to increase parallelism:
More independent instructions (ILP) within a thread (warp)
More concurrent threads (warps)
© NVIDIA 2013
Independent Instructions: ILP vs. TLP
SMX can leverage available Instruction-Level Parallelism more or
less interchangeably with Thread-Level Parallelism

Sometimes easier to increase ILP than to increase TLP
E.g., # of threads may be limited by algorithm or by HW resource limits
But if each thread has some degree of independent operations to do,
Kepler SMX can leverage that.  (E.g., a small loop that is unrolled.)

In fact, some degree of ILP is actually required to approach
theoretical max Instructions Per Clock (IPC)
© NVIDIA 2013
Control Flow
Instructions are issued per 32 threads (warp)

Divergent branches:
Threads within a single warp take different paths
if-else, ...
Different execution paths within a warp are serialized

Different warps can execute different code with no impact on
performance
© NVIDIA 2013
Control Flow
Avoid diverging within a warp
Note: some divergence is not necessarily a problem, but large
amounts impacts execution efficiency

Example with divergence:
if (threadIdx.x > 2) {...} else {...}
Branch granularity < warp size

Example without divergence:
if (threadIdx.x / warpSize > 2) {...} else {...}
Branch granularity is a whole multiple of warp size
© NVIDIA 2013
Control Flow
if ( ... )
{
     // then-clause

}
else
{
    // else-clause

}

instructions
© NVIDIA 2013
Execution within warps is coherent
instructions / time
Warp
(“vector” of threads)
35
34
33
63
62
32
3
2
1
31
30
0
Warp
(“vector” of threads)
© NVIDIA 2013
Execution diverges within a warp
instructions / time
3
2
1
31
30
0
35
34
33
63
62
32
© NVIDIA 2013
Execution diverges within a warp
instructions / time
3
2
1
31
30
0
35
34
33
63
62
32
Solution: Group threads with similar control flow
© NVIDIA 2013
Runtime Math Library and Intrinsics
Two types of runtime math library functions
__func(): many map directly to hardware ISA
Fast but lower accuracy (see CUDA Programming Guide for full details)
Examples: __sinf(x), __expf(x), __powf(x, y)
func(): compile to multiple instructions
Slower but higher accuracy (5 ulp or less)
Examples: sin(x), exp(x), pow(x, y)

A number of additional intrinsics:
__sincosf(), __frcp_rz(), ...
Explicit IEEE rounding modes (rz,rn,ru,rd)
© NVIDIA 2013
OPTIMIZE
Optimizing CPU-GPU Interaction: Maximizing PCIe Throughput
© NVIDIA 2013
Maximizing PCIe Throughput
Use transfers that are of reasonable size (a few MB, at least)
Use pinned system memory
Overlap memcopies with useful computation
© NVIDIA 2013
Pinned (non-pageable) memory
Pinned memory enables:
faster PCIe copies
memcopies asynchronous with CPU
memcopies asynchronous with GPU
Usage
cudaHostAlloc / cudaFreeHost
instead of malloc / free
cudaHostRegister / cudaHostUnregister
pin regular memory after allocation
Implication:
pinned memory is essentially removed from host virtual memory

© NVIDIA 2013
Asynchronicity in CUDA
Default:
Kernel launches are asynchronous with CPU
Memcopies (D2H, H2D) block CPU thread
CUDA calls are serialized by the driver
Streams and async functions provide additional asynchronicity:
Memcopies (D2H, H2D) asynchronous with CPU
Ability to concurrently execute kernels and memcopies

Stream: sequence of ops that execute in issue-order on GPU
Operations from different streams may be interleaved
Kernels and memcopies from different streams can be overlapped
© NVIDIA 2013
OPTIMIZE
Optimizing CPU-GPU Interaction: Overlapping Kernel
Execution with Memory Copies
© NVIDIA 2013
Overlap kernel and memory copy
Requirements:
D2H or H2D memcopy from pinned memory
Kernel and memcopy in different, non-0 streams
Code:
cudaStream_t   stream1, stream2;
cudaStreamCreate(&stream1);
cudaStreamCreate(&stream2);

cudaMemcpyAsync( dst, src, size, dir, stream1 );
kernel<<<grid, block, 0, stream2>>>(…);
potentially
overlapped
© NVIDIA 2013
Call Sequencing for Optimal Overlap
CUDA calls are dispatched in the sequence they were issued
Kepler can concurrently execute:
Up to 32 kernels
Up to 2 memcopies, as long as they are in different directions (D2H, H2D)
A call is dispatched if both are true:
Resources are available
Preceding calls in the same stream have completed
Scheduling:
Kernels are executed in the order in which they were issued
Thread blocks for a given kernel are scheduled if all thread blocks for
preceding kernels have been scheduled and SM resources still available
© NVIDIA 2013
Hyper-Q Enables Efficient Scheduling
Grid Management Unit selects most appropriate task from up to
32 hardware queues (CUDA streams)

Improves scheduling of concurrently executed grids

Particularly interesting for MPI applications when combined with
CUDA MPS (though not limited to MPI applications)
© NVIDIA 2013
Stream Examples without Hyper-Q
K1,M1,K2,M2: K1
K1 M1
M1 K2
K2 M2
M2
K1,K2,M1,M2: K1
K1 M1
M1 K2
K2 M2
M2
K1,M1,M2: K1
K1 M1
M1 M2
M2
K1,M2,M1: K1
K1 M1
M1 M2
M2
K1,M2,M2: K1
K1 M2
M2 M2
M2
Time
K:
Kernel
M:
Memcopy
Integer: Stream ID
© NVIDIA 2013
Stream Examples with Hyper-Q
K1,M1,K2,M2: K1
K1 M1
M1 K2
K2 M2
M2
K1,K2,M1,M2: K1
K1 M1
M1 K2
K2 M2
M2
K1,M1,M2: K1
K1 M1
M1 M2
M2
K1,M2,M1: K1
K1 M1
M1 M2
M2
K1,M2,M2: K1
K1 M2
M2 M2
M2
Time
K:
Kernel
M:
Memcopy
Integer: Stream ID
© NVIDIA 2013
Grid Management
Work Distributor
32 active grids
Stream Queue Mgmt
C
B
A
R
Q
P
Z
Y
X
Grid Management Unit
Pending & Suspended Grids
1000s of pending grids
SMX
SMX
SMX
SMX
SM
SM
SM
SM
Work Distributor
16 active grids
Stream Queue Mgmt
C
B
A
Z
Y
X
R
Q
P
CUDA
Generated
Work
Fermi
Kepler GK110
© NVIDIA 2013
Stream Dependencies Example
void foo(void)
{
    kernel_A<<<g,b,s, stream_1>>>();
    kernel_B<<<g,b,s, stream_1>>>();
    kernel_C<<<g,b,s, stream_1>>>();
}

void bar(void)
{
    kernel_P<<<g,b,s, stream_2>>>();
    kernel_Q<<<g,b,s, stream_2>>>();
    kernel_R<<<g,b,s, stream_2>>>();
}
stream_1
kernel_A
kernel_B
kernel_C
stream_2
kernel_P
kernel_Q
kernel_R
© NVIDIA 2013
Stream Dependencies without Hyper-Q
stream_1
kernel_A
kernel_B
kernel_C
stream_2
kernel_P
kernel_Q
kernel_R
Hardware Work Queue
R—Q—P     C—B—A
© NVIDIA 2013
Stream Dependencies with Hyper-Q
Hyper-Q allows 32-way concurrency
Avoids inter-stream dependencies

stream_1
kernel_A
kernel_B
kernel_C
stream_2
kernel_P
kernel_Q
kernel_R
C—B—A
R—Q—P
Multiple Hardware Work Queues
© NVIDIA 2013
Heterogeneous system: overlap work and data movement
Kepler + CUDA 5: Hyper-Q and CPU Callbacks
Hyper-Q Example: Building a Pipeline
DMA
DMA
© NVIDIA 2013
Tick-Tock Matrix Multiply
cudaMemcpyAsync(devA1, A[tile0], N, stream1);
cudaMemcpyAsync(devB1, B[tile0], N, stream1);
DGEMM<<<g,b,s, stream1>>>(devA1, devB1, devC1);

cudaMemcpyAsync(devA2, A[tile1], N, stream2);
cudaMemcpyAsync(devB2, B[tile1], N, stream2);
DGEMM<<<g,b,s, stream2>>>(devTileA, devTileB, devC1);

cudaMemcpyAsync(C[tile0], devC,  N, D2H, stream1);
cudaMemcpyAsync(devA1, A[tile2], N, H2D, stream1)
cudaMemcpyAsync(devB1, B[tile2], N, D2H, stream1)
DGEMM<<<g,b,s, stream1>>>(devA1, devB1, devC1);

cudaMemcpyAsync(C[tile1], devC,  N, D2H, stream1);
cudaMemcpyAsync(devA1, A[tile4], N, H2D, stream1);
cudaMemcpyAsync(devB1, B[tile4], N, D2H, stream1);
DGEMM<<<g,b,s, stream1>>>(devA1, devB1, devC1);
© NVIDIA 2013
Tick-Tock Matrix Multiply
dA1
stream 1
stream 2
memcpy

B[0]
dB1
DGEMM
dC_1 =dA_1 x dB_1
dC1
dA1
dB1
A[0]
C[0]
B[2]
A[2]
DGEMM
dC_2 =dA_2 x dB_2
DGEMM
C_1 =A_1 x B_1
B[1]
A[1]
dA2
dB2
memcpy

DGEMM
C_2 =A_2 x B_2
dC2
dA2
dB2
C[1]
B[3]
A[3]
C[2]
B[4]
A[4]
dC1
dA1
dB1
DGEMM
C_1 =A_1 x B_1
Copy Tile 0
Copy Tile 1
Compute Tile 0
Copy Tile 2
Compute Tile 1
Copy Tile 3
Compute Tile 2
Copy Tile 4
Compute Tile 3
GPU Memory
CPU Memory
memcpy

memcpy

memcpy

memcpy

dC2
dA2
dB2
C[3]
B[5]
A[5]
Copy Tile 5
Compute Tile 4
© NVIDIA 2013
Just a Higher Level of Parallelism
Problem is decomposed into parallel
“workers”.
At any given time
1 worker is using compute resources
1 worker is using copy transfers
Importantly:
The PCI-E link is kept saturated with
useful work.
For DGEMM, compute is also saturated.
Arch specific balancing
Depends on CPU and GPU
characteristics.
tile computed by stream 1
tile computed by stream 2
Result Matrix:
© NVIDIA 2013
Pipeline Code
for (unsigned int i = 0 ; i < nIterations ; ++i)
{
    // Copy data from host to device
    cudaMemcpyAsync(d_data, h_data, cpybytes, cudaMemcpyHostToDevice,
                    *r_streams.active());

    // Launch device kernel A
    kernel_A<<<gdim, bdim, 0, *r_streams.active()>>>();

    // Copy data from device to host
    cudaMemcpyAsync(h_data, d_data, cpybytes, cudaMemcpyDeviceToHost,
                    *r_streams.active());

    // Launch host post-process
    cudaStreamAddCallback(*r_streams.active(), cpu_callback,
                          r_streamids.active(), 0);

    // Rotate streams
    r_streams.rotate(); r_streamids.rotate();
}
© NVIDIA 2013
False dependencies prevent overlap
Breadth-first launch gives overlap, requires more complex code
Pipeline Without Hyper-Q
© NVIDIA 2013
Full overlap of all engines
Simple to program
Pipeline With Hyper-Q
© NVIDIA 2013
Hyper-Q also enables CUDA MPS
No application modifications necessary
Start MPS daemon using nvidia_cuda_mps_control -d
CUDA driver detects daemon and routes GPU accesses through it

Combines requests from several processes into one GPU context
(shared virtual memory space, concurrent kernels possible, etc.)

Allows for overlap of kernels with memcopies without explicit
use of streams
© NVIDIA 2013
But Hyper-Q != CUDA MPS
One process: No MPS required!
Automatically utilized
One or many host threads no problem
Just need multiple CUDA streams
Removes false dependencies among CUDA streams that
reduce effective concurrency on earlier GPUs

Multi-process: Use CUDA MPS
Leverages task-level parallelism across processes (e.g., MPI ranks)
MPI is not required for MPS – it’s just the common case for HPC
© NVIDIA 2013
Deploy
We’ve removed (or reduced) some bottleneck
Our app is now faster while remaining fully functional*
Let’s take advantage of that!

*Don’t forget to check correctness at every step

© NVIDIA 2013
GPU Optimization Fundamentals
Recap:
Develop systematically with APOD
Expose sufficient parallelism
Utilize parallel processing resources efficiently
Assess
Parallelize
Optimize
Deploy
© NVIDIA 2013
 Online Resources
www.udacity.com
docs.nvidia.com
developer.nvidia.com
devtalk.nvidia.com
www.stackoverflow.com


Skip links

DeepSpeed Inference: Multi-GPU inference with customized inference kernels and quantization support

March 15, 2021

Contents

While DeepSpeed supports training advanced large-scale models, using these trained models in the desired application scenarios is still challenging due to three major limitations in existing inference solutions: 1) lack of support for multi-GPU inference to fit large models and meet latency requirements, 2) limited GPU kernel performance when running inference with small batch sizes, and 3) difficulties in exploiting quantization, which includes both quantizing the model to reduce the model size and latency as well as supporting high-performance inference of quantized models without specialized hardware.

To handle these challenges, we introduce DeepSpeed Inference, which seamlessly adds high-performance inference support to large models trained in DeepSpeed with three key features:  inference-adapted parallelism for multi-GPU inference, inference-optimized kernels tuned for small batch sizes, and flexible support for quantize-aware training and inference kernels for quantized models.

Multi-GPU Inference with Adaptive Parallelism

Parallelism is an effective approach to fit large models and reduce per-device memory consumption for both training and inference. However, simply applying training parallelism choices and degree to inference does not work well. The MP and PP configuration is normally set during the model training, apart from the data parallelism (DP), based on the memory footprint and computation style, and resource budget. On one hand, inference computation intrinsically requires less memory, so it can afford a larger partition per device. It helps reduce the degree of parallelism needed for model deployment. On the other hand, optimizing latency or meeting latency requirements is often a first-class citizen in inference while training optimizes throughput.

To obtain desired latency, DeepSpeed Inference automatically adapts MP as an effective approach to reduce model latency, and its parallelism degree is often determined first. With MP, we can split the mode and parallelize computational operations across multiple devices (GPUs) to reduce latency, but it reduces computation granularity and increases communication that may hurt throughput. Once the latency target has been met, DeepSpeed can apply pipeline parallelism to maximize the throughput. Overall, DeepSpeed Inference supports flexible adaptation of both parallelism approach and degree choices from training to inference, minimizing latency while saving deployment costs.

Customized Inference Kernels for Boosted Compute Efficiency of Transformer Blocks

To achieve high compute efficiency, DeepSpeed-inference offers inference kernels tailored for Transformer blocks through operator fusion, taking model-parallelism for multi-GPU into account. The main difference between our kernel-fusion scheme and similar approaches is that we not only fuse element-wise operations (such as bias-add, residual, and activation function), but also merge the General matrix multiply (GeMM) operations with other operations. To do this, we design an efficient implementation for the vector-matrix or skinny matrix-matrix multiplication that allows us to fuse more operations at the reduction boundary of GeMM operations.

Kernel-Fusion

We take two main policies for fusing operations: 1) keeping the access-pattern of inputs and outputs intact throughout the sequence of operations fused together; 2) fusing operations at each all-reduce boundary. The first policy ensures that different thread-blocks won’t encounter transferring data between Streaming-Multiprocessors (SMs). This is due to no straight-forward communication among SMs other than using the main memory which adds the block-synching overhead because of non-deterministic behavior of memory access. The reason behind the second policy is that we cannot continue the execution unless the partial results are reduced among the model-parallel GPUs.

Figure 1: Transformer Layer with Megatron-style model-parallelism all-reduce components. The figure illustrates the parts of layer fused together with broken lines (width of line shows the fusion depth).

Figure 1 shows the different components of a Transformer layer, and the groups of operations considered for fusion in our inference optimization. We also consider the NVIDIA Megatron-LM style of parallelism that partitions attention (Attn) and feed-forward (FF) blocks across multiple GPUs. Thus, we include the two all-reduce operations that reduce the results among parallel GPUs after Attn and FF blocks. As Figure 1 shows, we fuse the operations inside a Transformer layer at four main regions:

To fuse these operations, we exploit shared-memory as an intermediate cache for transferring data between reduction operations used in layer-norm and GeMM, and the element-wise operations. Moreover, we use the warp-level instructions to communicate data between threads when reducing partial computations. In addition, we use a new schedule for GeMM operations, which allows for fusing as many operations as needed for the third kernel-fusion. We also combine the GeMMs for the attention computation in the second kernel-fusion, by using an implicit matrix transformation in order to reduce the memory pressure. Compared to the unfused computation style using cuBLAS GeMM, we improve the performance by 1.5x, 2.9x. 3x, and 1.2x for all these kernel-fusions, respectively.

Seamless pipeline from training to inference with automatic kernel-injection

To run the model in Inference mode, DeepSpeed simply requires the location of the model checkpoints and the desired parallelism configuration, i.e., MP/PP degree. DeepSpeed Inference kernels can also be enabled for many well-known model architectures such as HuggingFace (Bert and GPT-2) or Megatron GPT-based models using a pre-defined policy map that maps the original parameters to the parameters in the inference kernels. For other transformer-based models, user can specify their own policy map. Note that DS-Inference can run independent of the training pipeline as long as it receives all model checkpoints, and the DeepSpeed Transformer kernels for inference can be injected into any Transformer model if the right mapping policy is defined. For more information on how to enable Transformer inference kernel as well as specifying parallelism, please refer to out inference tutorial.

Flexible quantization support

To further reduce the inference cost for large-scale models, we created the DeepSpeed Quantization Toolkit, supporting flexible quantize-aware training and high-performance kernels for quantized inference.

For training, we introduce a novel approach called Mixture of Quantization (MoQ), which is inspired by mixed-precision training while seamlessly applying quantization. With MoQ, we can control the precision of the model by simulating the impact of quantization when updating the parameters at each step of training. Moreover, it supports flexible quantization policies and schedules—we find that by dynamically adjusting the number of quantization bits during training, the final quantized model provides higher accuracy under the same compression ratio. To adapt to different tasks, MoQ can also leverage the second order information of models to detect their sensitivity to precision and adjust the quantization schedule and target accordingly.

To maximize the performance gains from the quantization model, we provide inference kernels tailored for quantized models that reduce latency through optimizing data movement but do not require specialized hardware. Finally, our toolkit does not require any code changes on the client side, making it easy to use.

Performance results

Boosting throughput and reducing inference cost.  Figure 3 shows the inference throughput per GPU for the three model sizes corresponding to the three Transformer networks, GPT-2, Turing-NLG, and GPT-3. DeepSpeed Inference increases in per-GPU throughput by 2 to 4 times when using the same precision of FP16 as the baseline.  By enabling quantization, we boost throughput further. We reach a throughput improvement of 3x for GPT-2, 5x for Turing-NLG, and 3x for a model that is similar in characteristics and size to GPT-3, which directly translates to 3–5x inference cost reduction on serving these large models. In addition, we achieve these throughput and cost improvements without compromising latency as shown in Figure 5.

Figure 3: Inference throughput for different model sizes. DeepSpeed Inference achieves 3x to 5x higher throughput than baseline.

One source of inference cost reduction is through reducing the number of GPUs for hosting large models as shown in Figure 4.  The optimized GPU resources comes from 1) using inference-adapted parallelism, allowing users to adjust the model and pipeline parallelism degree from the trained model checkpoints, and 2) shrinking model memory footprint by half with INT8 quantization.  As shown in this figure, we use 2x less GPUs to run inference for the 17B model size by adapting the parallelism.  Together with INT8 quantization through DeepSpeed MoQ, we use 4x and 2x fewer GPUs for 17B and 175B sizes respectively.

Figure 4: Number of GPUs used for running inference on the different model sizes shown in Figure 4.

Reducing inference latency.  For the application scenarios where inference latency is critical, we can increase model parallelism degree in DeepSpeed Inference to reduce inference latency further.  As Figure 5 depicts, we can reduce the latency by 2.3x compared to PyTorch as we increase the model-parallelism size to 4.  Furthermore, we can still have high latency improvement with a fewer number of GPUs by adapting the parallelism at inference and using MoQ to quantize the model. We obtain 1.3x and 1.9x speedups while using 4x and 2x lower resources than baseline, respectively.

For the application scenarios where inference latency is critical, we can increase model parallelism degree in DeepSpeed Inference to reduce inference latency further.  As Figure 5 depicts, we can reduce the latency by 2.3x compared to PyTorch as we increase the model-parallelism size to 4.  Furthermore, we can still have high latency improvement with a fewer number of GPUs by adapting the parallelism at inference and using MoQ to quantize the model. We obtain 1.3x and 1.9x speedups while using 4x and 2x lower resources than baseline, respectively.

Figure 5. Inference latency for the 17B model using different parallelism configuration to optimize latency.

Updated: March 15, 2021


Source: Image created by Generative AI Lab using image generation models.

Maximizing GPU Kernel Optimization in Python with Triton

Author(s): Chaim Rand

TL;DR: Learn how to optimize your Python code for GPU using Triton. This book provides practical tips and techniques for improving performance and unleashing the full potential of GPU kernels. From data management to parallelization, it covers everything you need to know to master GPU kernel optimization in Python.”

Disclaimer: This post has been created automatically using generative AI. Including DALL-E, Gemini, OpenAI and others. Please take its contents with a grain of salt. For feedback on how we can improve, please email us

Introduction to Triton and GPU Kernel Optimization

In recent years, the use of graphics processing units (GPUs) has become increasingly popular in the field of data analysis and scientific computing. These powerful processors are capable of performing complex calculations and handling large datasets at lightning-fast speeds. However, harnessing the full potential of GPUs requires specialized knowledge and skills in optimization techniques. This is where Triton comes in – a powerful tool for GPU kernel optimization in Python and C++.

Understanding Triton and Its Capabilities

Triton is an open-source library developed by NVIDIA that allows users to write high-performance GPU kernels in Python and C++. It provides a simple and intuitive interface for writing code that can be executed on GPUs, without the need for complex and time-consuming low-level programming. With Triton, users can easily harness the full power of GPUs and accelerate their code, making it ideal for tasks such as machine learning, data analysis, and scientific simulations.

The Benefits of Using Triton for GPU Kernel Optimization

One of the main advantages of using Triton for GPU kernel optimization is its ease of use. With its simple and intuitive interface, even users with little or no experience in GPU programming can quickly learn how to write efficient and high-performing code. Additionally, Triton offers a wide range of built-in functions and optimizations that can significantly speed up the execution of code on GPUs. This not only saves time and effort but also allows users to focus on the logic and algorithms of their code rather than worrying about low-level optimizations.

Mastering GPU Kernel Optimization with Triton

To fully unleash the power of Triton, it is essential to understand its various optimization techniques and how to use them effectively. These include techniques such as data layout optimizations, loop unrolling, and memory coalescing, among others. Triton also provides a set of tools for profiling and debugging, which can help identify bottlenecks and optimize code further. By mastering these techniques and tools, users can achieve significant performance gains and fully utilize the capabilities of GPUs.

Real-World Applications of Triton in GPU Kernel Optimization

The applications of Triton in GPU kernel optimization are vast and diverse. From accelerating machine learning algorithms to speeding up scientific simulations, Triton has been used in a wide range of fields and industries. For example, researchers have used Triton to optimize code for computational fluid dynamics simulations, resulting in a 10x speedup compared to traditional CPU-based code. In the field of finance, Triton has been used to accelerate risk analysis calculations. With the increasing demand for faster and more powerful computing, understanding and utilizing GPU optimization techniques can be a valuable skill. With Triton, developers can easily harness the power of GPUs and achieve optimal results. It is a valuable tool for those looking to maximize their use of GPU technology in Python.

Crafted using generative AI from insights found on Towards Data Science.

Join us on this incredible generative AI journey and be a part of the revolution. Stay tuned for updates and insights on generative AI by following us on X or LinkedIn.

Introduction to Machine Learning, fourth edition (Adaptive Computation and Machine Learning series)

ChatGPT for Beginners Made Useful: Master the Fundamentals, Learn Useful Prompts, Boost Your Productivity and Build Passive Income (Generative AI & Chat GPT Mastery Series)

Super Study Guide: Transformers & Large Language Models

Disclaimer: The content on this website reflects the views of contributing authors and not necessarily those of Generative AI Lab. This site may contain sponsored content, affiliate links, and material created with generative AI. Thank you for your support.

Must read

Empowering Biology with Generative AI: GenBio AI’s Breakthrough

Generalizing Temporal Difference (TD) Algorithms with n-Step Bootstrapping in Reinforcement Learning

From Solo Notebooks to Collaborative Powerhouse: Essential VS Code Extensions for Data Science and Machine Learning Teams

Data Scientists Beware: The Power of Polars Over Pandas

More articles

Generalizing Temporal Difference (TD) Algorithms with n-Step Bootstrapping in Reinforcement Learning

From Solo Notebooks to Collaborative Powerhouse: Essential VS Code Extensions for Data Science and Machine Learning Teams

Data Scientists Beware: The Power of Polars Over Pandas

LEAVE A REPLY Cancel reply

Save my name, email, and website in this browser for the next time I comment.

Latest articles

Empowering Biology with Generative AI: GenBio AI’s Breakthrough

Generalizing Temporal Difference (TD) Algorithms with n-Step Bootstrapping in Reinforcement Learning

From Solo Notebooks to Collaborative Powerhouse: Essential VS Code Extensions for Data Science and Machine Learning Teams

Data Scientists Beware: The Power of Polars Over Pandas

Beyond LLMs: Compound Systems, Agents, and Building AI Products

About Us

Popular Category

Editor Picks

Best Books on Generative AI

Top Books on Large Language Models (LLMs)

©  Generative AI Lab Co.


Optimize TensorFlow GPU performance with the TensorFlow Profiler

Overview

This guide will show you how to use the TensorFlow Profiler with TensorBoard to
gain insight into and get the maximum performance out of your GPUs, and debug
when one or more of your GPUs are underutilized.

If you are new to the Profiler:

Keep in mind that offloading computations to GPU may not always be beneficial,
particularly for small models. There can be overhead due to:

Performance optimization workflow

This guide outlines how to debug performance issues starting with a single GPU,
then moving to a single host with multiple GPUs.

It is recommended to debug performance issues in the following order:

For example, if you are using a TensorFlow
distribution strategy
to train a model on a single host with multiple GPUs and notice suboptimal GPU
utilization, you should first optimize and debug the performance for one GPU
before debugging the multi-GPU system.

As a baseline for getting performant code on GPUs, this guide assumes you are
already using tf.function. The Keras Model.compile and Model.fit APIs will
utilize tf.function automatically under the hood. When writing a custom
training loop with tf.GradientTape, refer to the
Better performance with tf.function
on how to enable tf.functions.

The next sections discuss suggested approaches for each of the scenarios above
to help identify and fix performance bottlenecks.

1. Optimize the performance on one GPU

In an ideal case, your program should have high GPU utilization, minimal CPU
(the host) to GPU (the device) communication, and no overhead from the input
pipeline.

The first step in analyzing the performance is to get a profile for a model
running with one GPU.

TensorBoard's Profiler
overview page—which
shows a top level view of how your model performed during a profile run—can
provide an idea of how far away your program is from the ideal scenario.

The key numbers to pay attention to the overview page are:

Achieving optimal performance means maximizing these numbers in all three cases.
To get an in-depth understanding of your program, you will need to be familiar
with TensorBoard's Profiler
trace viewer. The
sections below show some common trace viewer patterns that you should look for
when diagnosing performance bottlenecks.

Below is an image of a model trace view running on one GPU. From the TensorFlow
Name Scope and TensorFlow Ops sections, you can identify different parts of
the model, like the forward pass, the loss function, backward pass/gradient
calculation, and the optimizer weight update. You can also have the ops running
on the GPU next to each Stream, which refer to CUDA streams. Each stream is
used for specific tasks. In this trace, Stream#118 is used to launch compute
kernels and device-to-device copies. Stream#119 is used for host-to-device
copy and Stream#120 for device to host copy.

The trace below shows common characteristics of a performant model.

For example, the GPU compute timeline (Stream#118) looks "busy" with very few
gaps. There are minimal copies from host to device (Stream #119) and from
device to host (Stream #120), as well as minimal gaps between steps. When you
run the Profiler for your program, you may not be able to identify these ideal
characteristics in your trace view. The rest of this guide covers common
scenarios and how to fix them.

1. Debug the input pipeline

The first step in GPU performance debugging is to determine if your program is
input-bound. The easiest way to figure this out is to use the Profiler’s
Input-pipeline analyzer,
on TensorBoard, which provides an overview of time spent in the input pipeline.

You can take the following potential actions if your input-pipeline contributes
significantly to step time:

In addition, refer to the
best practices for optimizing the input data pipeline.

2. Debug the performance of one GPU

There are several factors that can contribute to low GPU utilization. Below are
some scenarios commonly observed when looking at the
trace viewer and
potential solutions.

1. Analyze gaps between steps

A common observation when your program is not running optimally is gaps between
training steps. In the image of the trace view below, there is a large gap
between steps 8 and 9, meaning that the GPU is idle during that time.

If your trace viewer shows large gaps between steps, this could be an indication
that your program is input bound. In that case you should refer to the previous
section on debugging your input pipeline if you have not already done so.

However, even with an optimized input pipeline, you can still have gaps between
the end of one step and the start of another due to CPU thread contention.
tf.data makes use of background threads to parallelize pipeline processing.
These threads may interfere with GPU host-side activity that happens at the
beginning of each step, such as copying data or scheduling GPU operations.

If you notice large gaps on the host side, which schedules these ops on the GPU,
you can set the environment variable TF_GPU_THREAD_MODE=gpu_private. This
ensures that GPU kernels are launched from their own dedicated threads, and
don't get queued behind tf.data work.

Gaps between steps can also be caused by metric calculations, Keras callbacks,
or ops outside of tf.function that run on the host. These ops don’t have as
good performance as the ops inside a TensorFlow graph. Additionally, some of
these ops run on the CPU and copy tensors back and forth from the GPU.

If after optimizing your input pipeline you still notice gaps between steps in
the trace viewer, you should look at the model code between steps and check if
disabling callbacks/metrics improves performance. Some details of these ops are
also on the trace viewer (both device and host side).The recommendation in this
scenario is to amortize the overhead of these ops by executing them after a
fixed number of steps instead of every step. When using the Model.compile method in
the tf.keras API, setting the steps_per_execution flag does
this automatically. For custom training loops, use tf.while_loop.

2. Achieve higher device utilization

1. Small GPU kernels and host kernel launch delays

The host enqueues kernels to be run on the GPU, but there is a latency (around
20-40 μs) involved before kernels are actually executed on the GPU. In an ideal
case, the host enqueues enough kernels on the GPU such that the GPU spends most
of its time executing, rather than waiting on the host to enqueue more kernels.

The Profiler's
overview page on
TensorBoard shows how much time the GPU was idle due to waiting on the host to
launch kernels. In the image below, the GPU is idle for about 10% of the step
time waiting on kernels to be launched.

The trace viewer for
this same program shows small gaps between kernels where the host is busy
launching kernels on the GPU.

By launching a lot of small ops on the GPU (like a scalar add, for example), the
host might not keep up with the GPU. The
TensorFlow Stats
tool in TensorBoard for the same Profile shows 126,224 Mul operations taking
2.77 seconds. Thus, each kernel is about 21.9 μs, which is very small (around
the same time as launch latency) and can potentially result in host kernel
launch delays.

If your trace viewer
shows many small gaps between ops on the GPU like in the image above, you can:

2. TensorFlow op placement

The Profiler
overview page shows
you the percentage of ops placed on the host vs. the device (you can also verify
the placement of specific ops by looking at the
trace viewer. Like in
the image below, you want the percentage of ops on the host to be very small
compared to the device.

Ideally, most of the compute intensive ops should be placed on the GPU.

To find out which devices the operations and tensors in your model are assigned
to, set tf.debugging.set_log_device_placement(True) as the first statement of
your program.

Note that in some cases, even if you specify an op to be placed on a particular
device, its implementation might override this condition (example:tf.unique).
Even for single GPU training, specifying a distribution strategy, such as
tf.distribute.OneDeviceStrategy, can result in more deterministic placement of
ops on your device.

One reason for having the majority of ops placed on the GPU is to prevent
excessive memory copies between the host and the device (memory copies for model
input/output data between host and device are expected). An example of excessive
copying is demonstrated in the trace view below on GPU streams #167, #168,
and #169.

These copies can sometimes hurt the performance if they block GPU kernels from
executing. Memory copy operations in the
trace viewer have more
information about the ops that are the source of these copied tensors, but it
might not always be easy to associate a memCopy with an op. In these cases, it
is helpful to look at the ops nearby to check if the memory copy happens at the
same location in every step.

3. More efficient kernels on GPUs

Once your program's GPU utilization is acceptable, the next step is to look into
increasing the efficiency of the GPU kernels by utilizing Tensor Cores or fusing
ops.

1. Utilize Tensor Cores

Modern NVIDIA® GPUs have specialized
Tensor Cores that can
significantly improve the performance of eligible kernels.

You can use TensorBoard's
GPU kernel stats
to visualize which GPU kernels are Tensor Core-eligible, and which kernels are
using Tensor Cores. Enabling fp16 (see Enabling Mixed Precision section below)
is one way to make your program’s General Matrix Multiply (GEMM) kernels (matmul
ops) utilize the Tensor Core. GPU kernels use the Tensor Cores efficiently when
the precision is fp16 and input/output tensor dimensions are divisible by 8 or
16 (for int8).

For other detailed recommendations on how to make kernels efficient for GPUs,
refer to the
NVIDIA® deep learning performance
guide.

2. Fuse ops

Use tf.function(jit_compile=True) to fuse smaller ops to form bigger kernels
leading to significant performance gains. To learn more, refer to the
XLA guide.

3. Enable mixed precision and XLA

After following the above steps, enabling mixed precision and XLA are two
optional steps you can take to improve performance further. The suggested
approach is to enable them one by one and verify that the performance benefits
are as expected.

1. Enable mixed precision

The TensorFlow
Mixed precision guide
shows how to enable fp16 precision on GPUs. Enable
AMP on NVIDIA® GPUs to
use Tensor Cores and realize up to 3x overall speedups when compared to using
just fp32 (float32) precision on Volta and newer GPU architectures.

Make sure that matrix/tensor dimensions satisfy requirements for calling kernels
that use Tensor Cores. GPU kernels use the Tensor Cores efficiently when the
precision is fp16 and input/output dimensions are divisible by 8 or 16 (for
int8).

Note that with cuDNN v7.6.3 and later, convolution dimensions will automatically
be padded where necessary to leverage Tensor Cores.

Follow the best practices below to maximize the performance benefits of fp16
precision.

1. Use optimal fp16 kernels

With fp16 enabled, your program’s matrix multiplications (GEMM) kernels,
should use the corresponding fp16 version that utilizes the Tensor Cores.
However, in some cases, this does not happen and you do not experience the
expected speedup from enabling fp16, as your program falls back to the
inefficient implementation instead.

The GPU kernel
stats page shows which ops are Tensor Core eligible and which kernels are
actually using the efficient Tensor Core. The
NVIDIA® guide on deep learning performance
contains additional suggestions on how to leverage Tensor Cores. Additionally,
the benefits of using fp16 will also show in kernels that were previously
memory bound, as now the ops will take half the time.

2. Dynamic vs. static loss scaling

Loss scaling is necessary when using fp16 to prevent underflow due to low
precision. There are two types of loss scaling, dynamic and static, both of
which are explained in greater detail in the
Mixed Precision guide.
You can use the mixed_float16 policy to automatically enable loss scaling
within the Keras optimizer.

When trying to optimize performance, it is important to remember that dynamic
loss scaling can introduce additional conditional ops that run on the host, and
lead to gaps that will be visible between steps in the trace viewer. On the
other hand, static loss scaling does not have such overheads and can be a better
option in terms of performance with the catch that you need to specify the
correct static-loss scale value.

2. Enable XLA with tf.function(jit_compile=True) or auto-clustering

As a final step in getting the best performance with a single GPU, you can
experiment with enabling XLA, which will fuse ops and lead to better device
utilization and a lower memory footprint. For details on how to enable XLA in
your program with tf.function(jit_compile=True) or auto-clustering, refer to
the XLA guide.

You can set the global JIT level to -1 (off), 1, or 2. A higher level is
more aggressive and may reduce parallelism and use more memory. Set the value to
1 if you have memory restrictions. Note that XLA does not perform well for
models with variable input tensor shapes as the XLA compiler would have to keep
compiling kernels whenever it encounters new shapes.

2. Optimize the performance on the multi-GPU single host

The tf.distribute.MirroredStrategy API can be used to scale model training
from one GPU to multiple GPUs on a single host. (To learn more about how to do
distributed training with TensorFlow, refer to the
Distributed training with TensorFlow,
Use a GPU, and
Use TPUs guides and the
Distributed training with Keras
tutorial.)

Although the transition from one GPU to multiple GPUs should ideally be scalable
out of the box, you can sometimes encounter performance issues.

When going from training with a single GPU to multiple GPUs on the same host,
ideally you should experience the performance scaling with only the additional
overhead of gradient communication and increased host thread utilization.
Because of this overhead, you will not have an exact 2x speedup if you move from
1 to 2 GPUs, for example.

The trace view below shows an example of the extra communication overhead when
training on multiple GPUs. There is some overhead to concatenate the gradients,
communicate them across replicas, and split them before doing the weight update.

The following checklist will help you achieve better performance when optimizing
the performance in the multi-GPU scenario:

1. Optimize gradient AllReduce

When training with a synchronous strategy, each device receives a portion of the
input data.

After computing the forward and backwards passes through the model, the
gradients calculated on each device need to be aggregated and reduced. This
gradient AllReduce happens after the gradient calculation on each device, and
before the optimizer updates the model weights.

Each GPU first concatenates the gradients across the model layers, communicates
them across GPUs using tf.distribute.CrossDeviceOps
(tf.distribute.NcclAllReduce is the default), and then returns the gradients
after reduction per layer.

The optimizer will use these reduced gradients to update the weights of your
model. Ideally, this process should happen at the same time on all GPUs to
prevent any overheads.

The time to AllReduce should be approximately the same as:

(number of parameters * 4bytes)/ (communication bandwidth)

This calculation is useful as a quick check to understand whether the
performance you have when running a distributed training job is as expected, or
if you need to do further performance debugging. You can get the number of
parameters in your model from Model.summary.

Note that each model parameter is 4 bytes in size since TensorFlow uses fp32
(float32) to communicate gradients. Even when you have fp16 enabled, NCCL
AllReduce utilizes fp32 parameters.

To get the benefits of scaling, the step-time needs to be much higher compared
to these overheads. One way to achieve this is to use a higher batch size as
batch size affects step time, but does not impact the communication overhead.

2. GPU host thread contention

When running multiple GPUs, the CPU’s job is to keep all of the devices busy by
efficiently launching GPU kernels across the devices.

However, when there are a lot of independent operations that the CPU can
schedule on one GPU, the CPU can decide to use a lot of its host threads to keep
one GPU busy, and then launch kernels on another GPU in a non-deterministic
order. This can cause a skew or negative scaling, which can negatively affect
the performance.

The trace viewer below
shows the overhead when the CPU staggers GPU kernel launches inefficiently, as
GPU1 is idle and then starts running ops after GPU2 has started.

The trace view for the host shows that the host is launching kernels on GPU2
before launching them on GPU1 (note that the below tf_Compute* ops are not
indicative of CPU threads).

If you experience this kind of staggering of GPU kernels in your program’s trace
view, the recommended action is to:

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2022-09-15 UTC.

Stay connected

Support


CISC 879 : Software Support for Multicore Architectures
Yu, Xuan
Dept of Computer & Information Sciences
University of Delaware
Program Optimization Study on a
128-Core GPU
Shane Ryoo, Christopher I. Rodrigues, Sam S. Stone,
Sara S. Baghsorkhi, Sain-Zee Ueng, and Wen-mei W.
Hwu
CISC 879 : Software Support for Multicore Architectures
General Idea
Good news
Improving programmability and generality on
GPU
Possibility to perform a wide variety of
parallelization optimizations
Problem
How do you choose and control optimizations on
GPU properly

Possible combination of optimization is very large and
makes the optimization space tedious to explore

Limited local resources and global memory bandwidth
makes performance sensitive to even small changes in
code, unpredictable
CISC 879 : Software Support for Multicore Architectures
General Idea
• Presented a study that examines a
broad space of optimizations performed
on several applications
• Found configurations up to 74% faster
than previously thought optimal.
• Explained why this is happening on
GPU, discuss some principles and
techniques for finding near-optimal
configurations
CISC 879 : Software Support for Multicore Architectures
Organization
Architecture Overview (CUDA)

Introduction of execution hardware and threading model

Compute Unified Device Architecture (CUDA)
optimization space search

Discussion of the space search process and the classifications and
characteristics of the program optimizations
Experiments

Discuss result of the search for several applications

Matrix Multiplication

Magnetic resonance Imaging

Sums of Absolute Difference
Conclusion
CISC 879 : Software Support for Multicore Architectures
Architecture
• General Programming and compilation process
GPU is treaded as a coprocessor that
executes data-parallel kernel functions
The user supplies a both host (CPU) and
kernel (GPU) code
Codes are separated and compiled by
NVIDIA’s compiler.
Host code transfers data to GPU’s and
initialed the kernel code via API calls
CISC 879 : Software Support for Multicore Architectures
Architecture
16 streaming multiprocessors (SMs)
Each SM containing eight streaming
processors (SPs), or cores
Each core executes a single thread's
instruction in SIMD
multiply-add arithmetic unit
two special functional units (SFUs)
reciprocal square root, sine, and
cosine
fully pipelined,
CISC 879 : Software Support for Multicore Architectures
Architecture
CISC 879 : Software Support for Multicore Architectures
Architecture
Three Level Hierarchy:
•Grid
•Block
•Thread
Each kernel creates a single grid
A grid consists of many thread
blocks.(512, on single SM)
Threads in a block are organized
into warps of 32 threads. Each
warp executes in SIMD fashion,
issuing in four cycles on the eight
SPs of an SM.
When one wrap stall, SM switch to
another warp
CISC 879 : Software Support for Multicore Architectures
Architectural Interactions
• Hardware constraints
These constrains interacts with each other
making accurately predicting the effects of
one or more compiler optimizations of CUDA
difficult.
CISC 879 : Software Support for Multicore Architectures
Architectural Interactions
Consider an application:
•Uses 256 threads per block
•10 registers per thread
•4KB of shared memory per thread block.
Can schedule 3 thread blocks and 768 threads on each SM.
An optimization:
Increases each thread's register usage from 10 to 11 (an increase of only
10%) will decrease the number of blocks per SM from 3 to 2. This
decreases the number of threads on an SM by 33%.
Why?  768 * 11 = 8448 > 9192
CISC 879 : Software Support for Multicore Architectures
Architectural Interactions
By contrast, an optimization that increases
each thread block's shared memory usage by
1KB (an increase of 25%) does not decrease
the number of blocks per SM. Clearly, the
optimization space is inherently non-linear.
CISC 879 : Software Support for Multicore Architectures
Optimization space search
Architecture Overview (CUDA)

Introduction of execution hardware and threading model

Compute Unified Device Architecture (CUDA)
Optimization space search

Discussion of the space search process and the classifications and
characteristics of the program optimizations
Experiments

Discuss result of the search for several applications

Matrix Multiplication

Magnetic resonance Imaging

Sums of Absolute Difference
Conclusion
CISC 879 : Software Support for Multicore Architectures
Optimization space search
Basic strategy for good performance:
Reduce dynamic instruction count while maintaining high SP
occupancy.
Four categories of machine-level behavior to optimizae
Thread-level work redistribution
Instruction count reduction
Intra-thread parallelism
Resource balancing
CISC 879 : Software Support for Multicore Architectures
Example of matrix multiplication
The kernel is tiled so that each
thread block computes a square 16-
by-16 tile of the output matrix
Optimization space search
CISC 879 : Software Support for Multicore Architectures
Optimization space search
Example of matrix multiplication
tx and ty are each thread's coordinates
in the thread block;
indexA , indexB , and indexC are
positions in the matrices
Threads in a block cooperatively load
parts of the input matrices into shared
memory, amortizing the cost of global
load latency
Using larger tiles enhances the benefit
of data sharing, but reduces scheduling
flexibility since a greater fraction of the
threads on an SM must wait at barrier
synchronizations.
CISC 879 : Software Support for Multicore Architectures
Optimization space search
Four categories of machine-level behavior
to optimize
Thread-level work redistribution
Instruction count reduction
Intra-thread parallelism
Resource balancing
Each thread compute two matrix elements
instead of one, presents opportunities for
eliminating redundant instructions
previously distributed across threads
CISC 879 : Software Support for Multicore Architectures
Four categories of machine-level behavior
to optimize
Thread-level work redistribution
Instruction count reduction
Intra-thread parallelism
Resource balancing
Traditional compiler optimizations such as
common sub expression elimination, loop-
invariant code removal, and loop unrolling.
Optimization space search
CISC 879 : Software Support for Multicore Architectures
Four categories of machine-level behavior
to optimize
Thread-level work redistribution
Instruction count reduction
Intra-thread parallelism
Resource balancing
A developer can unroll loops to facilitate
code scheduling in the compiler or explicitly
insert pre-fetching code.
Optimization space search
CISC 879 : Software Support for Multicore Architectures
Four categories of machine-level behavior to optimize
Thread-level work redistribution
Instruction count reduction
Intra-thread parallelism
Resource balancing
Trade certain resource usages, some of which may be counterintuitive, to
produce a better performing application.
An example of this is using shared memory to buffer data for reuse,
regardless of whether it is shared with other threads.
Another example is proactive register spilling by the programmer. By
reducing register usage, often a critical resource, more thread blocks can be
assigned to each SM.
Optimization space search
CISC 879 : Software Support for Multicore Architectures
Experiments
Architecture Overview (CUDA)

Introduction of execution hardware and threading model

Compute Unified Device Architecture (CUDA)
optimization space search

Discussion of the space search process and the classifications and
characteristics of the program optimizations
Experiments

Discuss result of the search for several applications

Matrix Multiplication

Magnetic resonance Imaging

Sums of Absolute Difference
Conclusion
CISC 879 : Software Support for Multicore Architectures
Experiments
Comparison:
GPU experiments:
AMD Opteron 248 2.2GHz with 1GB main memory.
CPU versions:
Intel Core2 Extreme Quad running at 2.66 GHz with 4GB
main memory.
CISC 879 : Software Support for Multicore Architectures
Experiments
We varied tiling sizes, tiling dimensions, pre-fetching, and unroll factors,
CISC 879 : Software Support for Multicore Architectures
Experiments
The general trend: Larger tiles sizes and more work per thread gives
higher performance
Initial thought optimal
1x1 tiling, 16x16 tiles, complete unrolling, pre-fetching
87.5 GFLOPS.
Actual peak performing:
1x1 tiling, 16x16 tiles, complete unrolling, no pre-fetching
91.3 GFLOPS, an improvement of 4.2%.
CISC 879 : Software Support for Multicore Architectures
Experiments
Increasing the tiling dimensions:
•Stable performance
•Slight advantage in average
•does not result in peak performance.
Reason: negative effects of unrolling by more than a factor of two for the
higher tiling dimensions.
CISC 879 : Software Support for Multicore Architectures
Experiments
In summary:
larger thread blocks are good due to data sharing.
Complete unrolling often good due to reducing the branches calculations.
However, the runtime's scheduling may increase register pressure such that
the number of thread blocks assigned to each SM is reduced.
CISC 879 : Software Support for Multicore Architectures
Experiments
Another application:
Magnetic resonance imaging (MRI) reconstruction
Reconstruct high-quality images from non-Cartesian trajectories.
The computation required to perform these reconstructions is
substantial.
Parameters sensitive to performance:
loop unrolling factor,
The number of threads per block (tpb),
The number of scan points processed by each grid
CISC 879 : Software Support for Multicore Architectures
Shorter execution time for an unrolling factor of 8
4 is often worse than either 2 or 8
Reason:
•12 registers when unrolled, 2 thread blocks per SM and 6.17s.
•12 registers Unrolling factor is 2, 5.52s.
•24 registers unrolling factor is 4, only admit on block per SM 5.89s
•30 registers unrolling factor is 8, 1 block per SM, 4.64s
CISC 879 : Software Support for Multicore Architectures
Experiments
CISC 879 : Software Support for Multicore Architectures
there is a smaller chance of conflicts when fewer thread
blocks run on an SM.
Experiments
CISC 879 : Software Support for Multicore Architectures
To summarize MRI
Performance relatively insensitive to block size
An unrolling factor of 8 provided the highest performing
Experiments
CISC 879 : Software Support for Multicore Architectures
Gradual changes in optimization parameters can have wildly
varying effects on an application.
Local resources used by a thread increases to points where
fewer thread blocks can be assigned to each SM will reduce
overall performance.
They believe that scheduling should be better
controlled, possibly by the compiler rather than the runtime.
Conclusions