Received: 8 August 2019
Accepted: 28 August 2019
DOI: 10.1002/cpe.5547
S P E C I A L I S S U E P A P E R
Hierarchical Roofline analysis for GPUs: Accelerating
performance optimization for the NERSC-9 Perlmutter system
Charlene Yang1
Thorsten Kurth1
Samuel Williams2
1National Energy Research Scientific
Computing Center (NERSC), Lawrence
Berkeley National Laboratory, Berkeley,
California
2Computational Research Division (CRD),
Lawrence Berkeley National Laboratory,
Berkeley, California,
Correspondence
Charlene Yang, National Energy Research
Scientific Computing Center (NERSC),
Lawrence Berkeley National Laboratory,
Berkeley, CA 94720.
Email: cjyang@lbl.gov
Funding information
Advanced Scientific Computing Research
Program, Department of Energy, Office of
Science, U.S., Grant/Award Number:
DE-AC02-05CH11231; National Energy
Research Scientific Computing Center
(NERSC), which is supported by the Office of
Science of the U.S. Department of Energy,
Grant/Award Number: DE-AC02-05CH11231
Summary
The Roofline performance model provides an intuitive and insightful approach to identify-
ing performance bottlenecks and guiding performance optimization. In preparation for the
next-generation supercomputer Perlmutter at NERSC, this paper presents a methodology to
construct a hierarchical Roofline on NVIDIA GPUs and extends it to support reduced pre-
cision and Tensor Cores. The hierarchical Roofline incorporates L1, L2, device memory, and
system memory bandwidths into one single figure, and it offers more profound insights into
performance analysis than the traditional DRAM-only Roofline. We use our Roofline method-
ology to analyze three proxy applications: GPP from BerkeleyGW, HPGMG from AMReX, and
conv2d from TensorFlow. In doing so, we demonstrate the ability of our methodology to readily
understand various aspects of performance and performance bottlenecks on NVIDIA GPUs and
motivate code optimizations.
KEYWORDS
code optimization, Cray, NVIDIA GPU, performance analysis, Roofline, tensor core
1
INTRODUCTION
NERSC's next supercomputer Perlmutter will be an NVIDIA GPU-accelerated Cray supercomputer with AMD EPYC host CPUs and an
Ethernet-compatible Slingshot network. Although NERSC users are generally familiar with performance optimization on Intel and AMD CPUs,
there are a number of new facets of performance optimization on GPUs, including thread predication, deep memory hierarchies, mixed precision
computation, and Tensor Cores that need to be better understood.
Rather than forcing users to embrace a ‘‘trial-and-error’’ approach to performance optimization or dig through numerous profiler metrics, the
Roofline performance model1 provides a visually intuitive way for users to identify performance bottlenecks and motivate code optimization
strategies. Roofline is a throughput-oriented performance model centered around the interplay between computational capabilities (eg, peak
GFLOP/s), memory bandwidth (eg, STREAM GB/s), and data locality (ie, reuse of data once it is loaded from memory). Data locality is commonly
expressed as arithmetic intensity, which is the ratio of floating-point operations performed to data movement (FLOPs:Byte). Performance
(GFLOP/s) is bound by
GFLOP/s ≤min
⎧
⎪
⎨
⎪⎩
Peak GFLOP/s
Peak GB/s × Arithmetic Intensity,
(1)
which produces the traditional Roofline formulation when plotted on a log-log plot.
Previously, the Roofline model was expanded to support the full memory hierarchy2,3 by adding additional bandwidth ‘‘ceilings.’’ Similarly,
additional ceilings beneath the Roofline can be added to represent performance bottlenecks arising from lack of vectorization or the failure to
exploit fused multiply-add (FMA) instructions.
Published 2019. This article is a U.S. Government work and is in the public domain in the USA
Concurrency Computat Pract Exper. 2019;e5547.
wileyonlinelibrary.com/journal/cpe
1 of 12
https://doi.org/10.1002/cpe.5547
Published 2019. This article is a U.S. Government work and is in the public domain in the USA
Concurrency Computat Pract Exper. 2020;32:e5547.
wileyonlinelibrary.com/journal/cpe
1 of 12
https://doi.org/10.1002/cpe.5547
2 of 12
YANG ET AL.
Orthogonal to the Roofline description of hardware is characterizing applications in terms of Roofline-related coordinates, ie, Performance
(GFLOP/s) and Arithmetic Intensity (FLOPs/Byte). One can employ a variety of methods to calculate these terms ranging from hand counting
FLOPs and estimating bytes, to performance counters,4,5 to software simulators2 that trade performance for accuracy.
Over the last decade, Roofline analysis has been proven to be a great success, especially with the hierarchical Roofline on Intel CPUs2 and
was a benefit to understanding performance on NERSC's previous KNL-based Cori Supercomputer.6,7 However, Roofline has yet to be fully
developed on NVIDIA GPUs. This paper builds upon the previous work on CPU architectures as well as the HBM-only Roofline methodology
we developed for NVIDIA GPUs8 and expands the model into a hierarchical Roofline methodology that also captures the performance effects
associated with reduced precision and Tensor Cores on NVIDIA's latest V100 GPUs.
Our expanded methodology includes the following.
• Empirical measurement of peak performance (GFLOP/s) and bandwidth (GB/s);
• Accurate measurement of the total number of FLOPs in the code;
• Accurate measurement of data movement in the code, throughout the memory/cache hierarchy, ie, BytesL1, BytesL2, BytesHBM, BytesSystemMemory;
• Calculation of arithmetic intensities on various memory/cache levels, ie, FLOPs:ByteL1, FLOPs:ByteL2, FLOPs:ByteHBM, FLOPs:ByteSystemMemory;
• Quantifying the performance implications of FMA, FPADD, and FPMUL in the instruction mix;
• Quantifying the performance implications of reduced precision (FP16 and FP32) and Tensor Cores;
• Plotting application performance against architecture peaks.
In this paper, we provide a detailed description of our Roofline methodology for NVIDIA GPUs. We then apply this methodology to three
proxy applications, GPP from the Material Science code BerkeleyGW,9 HPGMG from the Adaptive Mesh Refinement framework AMReX,10 and
conv2d from TensorFlow.11 For each of these applications, we include multiple variants of the same code in order to highlight the ability of our
methodology to capture the different nuances of performance analysis on NVIDIA GPUs. Throughout this process, we provide a detailed analysis
of the information our Roofline methodology extracts. Finally, we conclude the paper with some high-level insights, observations, and espouse
several directions for future work.
2
ROOFLINE METHODOLOGY ON NVIDIA GPUS
In order to affect Roofline analysis of GPU-accelerated applications, one must perform three steps. First, one must characterize the underlying
GPU's computational capabilities in terms of this Roofline model. In effect, this is measuring peak performance (GFLOP/s) and bandwidth (GB/s)
as a function of data precision, operation, and memory/cache level. Second, one must characterize the execution of an application and extract
the relevant Roofline-related parameters, including data movement at each level of the memory hierarchy, floating-point operations performed
(by precision), and kernel run times. Finally, one must synthesize these two data sets together and plot them in a single figure.
2.1
Architectural characterization
The Empirical Roofline Toolkit (ERT)12 was developed to characterize multicore, manycore, and GPU-accelerated systems. It was written in
MPI+OpenMP+CUDA in order to replicate the most common programming environments on DOE (Department of Energy) supercomputers. ERT
defines a kernel of varying L1 arithmetic intensity on a parameterized vector. By sweeping a range of arithmetic intensities and vector sizes, it
can extract the peak performance of a target platform as well as the bandwidth at each level of the memory hierarchy. In this paper, we used the
MPI+CUDA implementation of ERT to characterize a single Volta V100 GPU. Unfortunately, ERT, as written, consistently fails to identify the L1
cache on NVIDIA GPUs. To that end, throughout this paper, we use a theoretical L1 bandwidth coupled with empirical (ERT) bandwidths for L2
and HBM, for the Roofline ceilings.
ERT, as written, is solely a double-precision benchmark. As such, in this paper, we use a simple linear extrapolation for single- and half-precision
performance. Moreover, whereas ERT's kernels are optimized for fused-multiply-add (FMA) instruction-set architectures, we estimate the penalty
of not exploiting FMA by defining a ‘‘no FMA’’ ceiling that is half the FMA performance. NVIDIA GPUs implement 16-bit (FP16) Tensor Core
matrix-matrix multiplications. Throughout this paper, we use the theoretical peak Tensor Core performance. This may seem optimistic, but we
will show it does not skew our analysis.
Ultimately, we collect 10 performance numbers for our target GPU: L1, L2, and HBM bandwidth, FP16/FP32/FP64 FMA and FP16/FP32/FP64
‘‘no FMA’’ performance, and Tensor Core peak performance. For brevity, Figure 1 plots measured ERT and theoretical performance for FP64
FMA, no-FMA, L2, and HBM on an NVIDIA Volta V100 GPU. Clearly, theoretical performance generally overestimates attainable performance
by about 10%.
2.2
Application characterization
In this paper, we leverage the proof of concept methodology developed by Yang et al8 and extend it to support both hierarchical (L1, L2, HBM,
System Memory) Roofline analysis as well as FP32 and FP16 precision (including Tensor Core). To that end, we use nvprof to collect a set of
2 of 12
YANG ET AL.
YANG ET AL.
3 of 12
FIGURE 1
NVIDIA V100 Hierarchical Roofline Ceilings. Observe
V100 advertised performance is very close to empirical performance
metrics for each kernel in an application. We then synthesize those metrics together in order to plot each kernel on a Roofline using its Arithmetic
Intensity (x) and GFLOP/s (y) coordinates.
In order to calculate a kernel's arithmetic intensity (AI) and GFLOP/s performance, we must collect three raw quantities, ie, kernel run time,
FLOPs executed (forFP64, FP32, and FP16), and bytes read and written by each level of the memory hierarchy (L1, L2, HBM, and System Memory).
AI<precision>,<level> = nvprof FLOPs<precision>
nvprof Bytes<level>
(2)
FLOP/s<precision> = nvprof FLOPs<precision>
nvprof Run Time
,
(3)
where <level> can be L1, L2, HBM (Device Memory) or System Memory, and <precision> can be FP64, FP32, FP16, or Tensor Core.
Kernel run time: To collect application run time, we use the following commands to obtain either the timing of a particular invocation of a
kernel or the average timing of a kernel over multiple invocations.
nvprof −−print −gpu −trace.∕application
nvprof −−print −gpu −summary.∕application.
Kernel FLOPs: nvprof provides a rich set of metrics to measure the total number of FLOPs executed in a kernel. These metrics only account for
nonpredicated threads, so operations that are masked out are not included. For complex operations such as divides, logarithms, and exponentials,
each operation is implemented with multiple instructions and hence is counted as multiple FLOPs. To collect the FLOP counts, we use
nvprof–kernels < kernel_name > –metrics < metric_name > ./application,
where <metric_name> can be flop_count_dp, flop_count_sp, or flop_count_hp for FP64, FP32, and FP16, respectively.
The aforementioned floating-point metrics can account for the majority of FLOPs in a large range of applications. However, FLOPs executed
inside the NVIDIA V100 Tensor Cores are not captured by these counters. The Tensor Cores are designed to accelerated matrix-FMA operations,
that is, operations of the form D = A ·B+C, where A, B are real valued 4×4 matrices in half precision (FP16) and C, D are real valued 4×4 matrices
in 16-bit (FP16) or 32-bit (FP32) precision. In the latter case, the accumulator in the Tensor Core operation is performed using FP32 arithmetic.
As of early 2019, nvprof does not offer an accurate flop_count_ metric for Tensor Cores, like for the normal SM cores but rather a
‘‘utilization’’ metric tensor_precision_fu_utilization. This metric spans an integer range from 0 (not used) to 10 (fully utilized). In order
to estimate the Tensor Core FLOP count, we assume that a utilization value of 10 corresponds to 125 TFLOP/s and then multiply this number
with the run time of the kernel to estimate the total number of FLOPs. It is expected that NVIDIA's next-generation Nsight profiling tool will
have enhanced capabilities to measure Tensor Core FLOPs, and we will investigate that in our future work.
Bytes: The data moved between each two levels in the memory/cache hierarchy must be collected in order to construct the hierarchical
Roofline. We use nvprof to collect the total number of read and write transactions and multiply the total by the size of each transaction in bytes
Bytes = (read transactions + write transactions) × transaction size.
(4)
YANG ET AL.
3 of 12
4 of 12
YANG ET AL.
TABLE 1
nvprof metrics for measuring data traffic in the memory/cache hierarchy
Level
Metrics
Transaction size
L1 cache
gld_transactions, gst_transactions, atomic_transactions,
32B
local_load_transactions, local_store_transactions,
shared_load_transactions, shared_store_transactions
L2 cache
l2_read_transactions, l2_write_transactions
32B
HBM memory
dram_read_transactions, dram_write_transactions
32B
PCIe/NVLINK
system_read_transactions, system_write_transactions
32B
The invocation of nvprof on command line is the same as when collecting FLOPs, but the metrics are more complicated (see Table 1)*. Note
that in this paper, all applications fit in the GPU's HBM memory. As such, system transactions are virtually zero as there is no data movement
over PCIe/NVLINK. Thus, they will not be presented in Section 4.
2.3
Roofline visualization
With both the empirical ceilings collected via ERT and the application kernel AI (2) and GFLOP/s (3) coordinates determined through the collection
of nvprof metrics, we plot the resultant Roofline model using Python Matplotlib13. Some of our Matplotlib scripts are available on GitHub,14
and users are free to tweak them based on their specific needs. The most basic example plot_roofline.py takes an input file that specifies
the memory ceilings, compute ceilings, AI's for each kernel, and GFLOP/s performance for each kernel.
3
EXPERIMENTAL SETUP
In this section, we describe our test machine, software configuration, and the applications we use to evaluate our hierarchical GPU Roofline
methodology.
3.1
Hardware and software configuration
Results presented in this paper were all obtained on the Cori supercomputer at the National Energy Research Scientific Center (NERSC),
Lawrence Berkeley National Laboratory (LBNL). To prepare for the arrival of the next-generation supercomputer Perlmutter, NERSC has installed
a GPU-accelerated partition on Cori, comprised of nodes with Intel Skylake CPUs and NVIDIA V100 GPUs (4:1 GPU:CPU ratio). This partition will
enable the NESAP (NERSC Exascale Science Applications Program) teams in their prototyping, debugging, porting, and development activities in
preparation for migrating to Perlmutter.
CUDA 10 is installed on Cori GPU partition and nvprof, is frequently used in this paper, and is part of the CUDA Toolkit. Moreover, ERT
is from the BitBucket repository12 with example configuration scripts deployed for the Cori GPU partition. For the conv2d benchmark from
TensorFlow, we used TensorFlow v1.12.0 linked against CUDA 9.0 and cuDNN 7.3.1.
3.2
Benchmarks
In this paper, we evaluate our Roofline methodology for NVIDIA GPUs using three benchmarks: GPP from BerkeleyGW, HPGMG from AMRex,
and conv2d from TensorFlow. These benchmarks were selected because they exhibit a range of computational characteristics, including a range
of memory access patterns, data types, data locality, and thread divergence properties.
GPP: The General Plasmon Pole (GPP) kernel9 is a proxy application based on the BerkeleyGW Material Science code.15 It calculates electron
self-energy using the common General Plasmon Pole approximation.16 GPP is written in C++ and accelerated with CUDA. The computation in the
kernel is tensor-contraction–like, wherein a few precalculated complex double-precision arrays are multiplied and summed over one dimension
and collapsed into a small matrix. The problem we chose in this paper is comprised of 512 electrons and 32 768 plane wave basis elements and is a
medium-sized problem for real-world materials science. It requires around 1.5GB of memory and fits well into the HBM memory on a V100 GPU.
The pseudocode of the GPP kernel can be described as
*Add surface and texture related metrics if surface and texture memory are in use, for example, in Graphics applications.
4 of 12
YANG ET AL.
YANG ET AL.
5 of 12
Not only does GPP offer abundant parallelism that can be mapped to threads, warps, and thread blocks, but it also is heavily parameterized
in order to capture the full spectrum of realistic problem configurations. One of these parameters, nw, enables arbitrary increases in arithmetic
intensity by increasing reuse of the arrays accessed within the iw loop. Similarly, we can modify the ig loop to affect strided memory accesses.
Ultimately, not only does this single well-understood kernel act as a stand-in for a range of potential application kernels, but it also enables us to
test the limits of our Roofline methodology for NVIDIA GPUs.
HPGMG GSRB smoother: HPGMG10,17,18 is a geometric multigrid benchmark designed to proxy the multigrid solves found in block structured
AMR (Adaptive Mesh Refinement) applications that use the AMReX framework.19 HPGMG solves the fourth-order variable-coefficient Laplacian
on a unit-cube with Dirichlet boundary conditions using a multigrid F-cycle. As such, it is only moderately compute–intensive (FP64 FLOPs:Byte>1),
but it is highly demanding of SIMT and cache locality. HPGMG was originally implemented in C with MPI and OpenMP parallelization and has
shown scalability to 8.5 million cores. It was subsequently extended to support GPUs on the finer (larger) mesh levels.10
The Gauss-Seidel Red-Black (GSRB) smoother generally dominates HPGMG's run time. However, the performance of the smoother varies
substantially among the various multigrid levels (mesh sizes). GSRB smoothers perform two stencil kernel invocations per smooth (red and black).
Cells are marked as either red or black in a 3D checkerboard pattern. Cells matching the sweep color are updated, while the others are simply
copied to the result array. Thus, for a 1283 box, one must perform 1283∕2 =1 M stencils per kernel invocation.
In order to balance the quality of a compiler against hardware's ability to efficiently execute strided memory access patterns or predication,
HPGMG includes three different implementations of its GSRB smoother (GSRB_FP, GSRB_BRANCH, and GSRB_STRIDE2). All three implemen-
tations perform the same computation and touch the same data over the course of a threadblock's execution. As such, they represent an ideal
testbed for using Roofline and performance tools to understand the subtle interactions between compiler and hardware. The GSRB_FP imple-
mentation realizes red-black updates via multiplication by a precomputed auxiliary array of 1.0's and 0.0's. Such an implementation is trivially
vectorized but requires twice the computation and an additional load from cache. The GSRB_BRANCH version uses a branch to perform stencils
only on cells whose color matches the sweep while copying data for the others. Such an implementation can be realized through either predication
(masking) or loop fissioning into two stride-2 updates. Regardless, the number of floating-point operations is minimized, but execution on SIMD or
SIMT hardware may preclude this. Finally, in the GSRB_STRIDE2 implementation, each CUDA thread is responsible for two adjacent cells within
a plane. The thread updates the cell whose color matches the sweep color and copies the other. This implementation minimizes computation,
avoids predication of computation, but requires the compiler/hardware to execute efficient stride-2 L1 cache accesses (HBM access will always
be coalesced into unit-stride transactions).
For this paper, we run HPGMG with eight 1283 boxes (./hpgmg-fv 7 8). This results in levels 5 to 8 running on GPU and levels 1 to 4 on
CPU. As this paper is focused on Roofline on GPUs, we only examine levels 5 to 8.
TensorFlow conv2d: TensorFlow11,20 is a deep learning framework that allows users to express complicated neural network graphs in a
reasonable amount of lines of code. Besides flexibility, it also provides performance portability across a variety of computing architectures by
using optimized libraries such as cuDNN. 2D convolution layers are the most compute-intensive kernels in most modern deep neural networks,
so in this paper, we chose to analyze tf.nn.conv2d21 for our Roofline validation.
In a forward pass, tf.nn.conv2d performs 2D convolution of an input tensor and a convolution kernel and produces another tensor as the
output. Assume an input tensor A of shape N × H × W × C, where N is the number of samples in the batch, H and W are the height and width,
and C is the number of channels. With a convolution kernel K of shape KH, KW, C, C′, the resultant tensor B is of shape N× H′ × W′ × C′, where
H′= H −KH + 1, W′= W −KW + 1, and the individual elements are given by
Bnhwc =
C−1
∑
m=0
KH−1
∑
kh=0
KW−1
∑
kw=0
An h+kh w+wh m Kkh kw m c.
(5)
In the more computationally expensive backward pass, the derivative of Equation (5) is computed with respect to K. The TensorFlow routine
further allows for generalizations such as nonunit stride or dilation. Values at the image boundary are treated separately, eg, by replication or
zero-padding.22 In this study, we will focus on the performance characteristics of a typical convolution kernel found in one of the most commonly
used networks for image analysis, ResNet50.23 We analyze this kernel for forward and backward passes as well as the effects of parameters such
as batch, input, kernel, and stride sizes on their performance.
Most TensorFlow routines support specifying different precisions via a dtype argument. For deep learning applications, the most relevant are
FP16 and FP32, and we will focus on these two precisions in this paper. The convolution operations such as those in Equation (5) are essentially
YANG ET AL.
5 of 12
6 of 12
YANG ET AL.
matrix-matrix multiplications, and on NVIDIA's Volta GPUs, TensorFlow can leverage the specialized instructions such as HMMA's on the Tensor
Cores for more performance. Due to this GEMM-like operation, the conv2d kernel will have a much higher arithmetic intensity than our previous
two examples, GPP and HPGMG.
The pseudocode of the conv2d kernel is as follows.
The main measurement part is toward the end of this code block (the last four lines). The pyc.driver commands are from PyCUDA,24 and
they allow for precise instrumentation on this region. To facilitate this, nvprof will also need to be launched with –profile-from-start
off to disable profiling from the start of the application.
Measuring the performance of a TensorFlow kernel is not straightforward. First, every kernel can translate into a series of subkernel calls.
These subkernels include the kernel that does the essential computation but also kernels that perform housekeeping work before and after the
main kernel such as data layout transformation, data type transformation, and index calculation. Second, the autotuning mechanism in TensorFlow
and cuDNN library adds to the complexity of the measurement of a kernel's performance. TensorFlow selects the best performing sequence of
subkernels based on input parameters and some heuristic data, and the cuDNN library, frequently called by TensorFlow, also performs certain
algorithm selection.
In this paper, we include the housekeeping subkernels in our performance measurement of conv2d as they are necessary to the core
computation subkernel, but we exclude the autotuning subkernels because we want to focus on the best implementation of the conv2d kernel
for given parameters. To achieve this, we split the loop in tf.Session into two parts: a warm-up part with trip count n_warm = 5 and a
measurement part with trip count n_iter = 20. Depending on the value of pass, exec_op will either compute the convolution (forward pass), or
the convolution along with its derivatives (backward pass). The extra calibrate option allows for exclusion of subkernels that are associated
with random input tensor generation, and we would like to exclude that in our measurement as well.
In Section 4, we use the sum of the following three measurements as the FLOP count of conv2d when calculating Arithmetic Intensity
and the GFLOP/s performance, flop_count_sp, flop_count_hp, and FLOPs derived from tensor_precision_fu_utilization (see
Section 2.2). Here, we include the flop_count_sp metric (metric for FP32 operations) even in the FP16 kernels because even if the input
and output tensors are set to be in FP16 precision, TensorFlow could still decide to deploy subkernels that are in FP32 precision based on its
autotuning mechanism. An example of this is the backward pass in Figure 10. However, all the other kernels in Figures 8, 9, and 10 execute as
expected, ie, FP16 kernels are run on the Tensor Core, and FP32 kernels are run on the common cores.
In this paper, if not otherwise stated, we use input tensor size 112 × 112 × 64 and kernel size 3 × 3 × 64 × C′, where C′ is the number of output
filters as defined in Equation (5), C′= 64, and the stride size is 2.
4
RESULTS
In this section, we discuss our observations and insights from applying our Roofline methodology for NVIDIA GPUs to our three benchmarks.
4.1
GPP performance analysis
Figure 2 presents a hierarchical Roofline model for GPP running on a V100 GPU as a function of the parameter nw. Recall that nw increases
arithmetic intensity at all levels of the memory hierarchy as it creates a tighter inner loop that reuses data loaded from the cache. We observe
several effects.
6 of 12
YANG ET AL.
YANG ET AL.
7 of 12
FIGURE 2
GPP hierarchical Roofline on the NVIDIA V100 GPU as a
function of nw (stride=1)
FIGURE 3
GPP Roofline on the NVIDIA V100 GPU as a function of
the FMA instruction mix. Note that 60% of GPP's floating-point
instructions are FMA
First, both L1 and HBM arithmetic intensity (x coordinate of each dot) increase linearly with nw. Interestingly, L2 intensity shows a much more
complex pattern. Linear increases in arithmetic intensity for kernels that are linearly increasing the number of FLOPs (numerator of arithmetic
intensity) implies roughly constant data movement (denominator of arithmetic intensity). As such, we can infer very good locality in the register
file (constant L1 intensity) as well as good locality in the L2 (constant HBM intensity). However, the only slight improvement in L2 intensity
implies substantial increases in L2 data movement or substantial losses in L1 locality.
Second, HBM intensity is consistently much larger than L1 intensity, implying that there is substantially higher locality in the caches than in
the register file. Moreover, L2 intensity is initially (nw = 1) very close to HBM intensity, but it approaches L1 intensity as nw approaches 6. This is
indicative of a transition from a regime where there is high L1 locality and virtually no L2 locality (nw = 1) to a regime where there is virtually no
L1 locality but high L2 locality (nw = 6).
Third, at low nw, performance is clearly bound by HBM (green curve tracks the HBM ceiling). However, as nw increases, performance quickly
saturates. Our hierarchical Roofline analysis demonstrates GPP is clearly not bound by either L1 or L2 bandwidth (red and blue curves are far
from their respective ceilings). This indicates that other effects have manifested that limit performance.
Unlike linear algebra routines (eg, matrix-multiplications), GPP includes a mix of floating-point adds, multiplies, and fused multiply-adds (FMA).
As 100% of the dynamic instructions are not FMA, peak performance will never be attainable.
In fact, we can create an effective ceiling by using nvprof to collect the number of FMA and non-FMA floating-point instructions. Using
Equation (6), it is observed that at nw = 6, only 60% of the floating-point instructions GPP executes are FMA. Moreover, Equation (7) bounds
GPP performance at 80% of the V100's (double-precision) FMA peak performance. However, Figure 3 shows that the observed performance
from GPP is roughly 66% of the full FMA peak. This clearly indicates that other aspects of GPU execution are ultimately limiting performance
𝛼=
FMA FP64 instr.
FMA FP64 instr. + non-FMA FP64 instr. = 60%
(6)
𝛽= 𝛼× 2 + (1 −𝛼)
2
= 80%.
(7)
As mentioned, GPP is highly parameterizable. To that end, Figure 4 shows a Roofline model for the strided implementation of GPP. Here,
threads within a warp update access every nth element (threads stride by 32 ∗n words instead of the nominal Stride-32). Unlike our previous
work,8 which focused solely on HBM Rooflines for GPUs, the GPP hierarchical Roofline shows that the L1 and L2 cache behave quite differently
from HBM. Whereas HBM intensity decreases linearly with increasing stride up to Stride-4 (4 double complex words = 64 Bytes), L1 and L2
intensity stops decreasing beyond Stride-2 (32B). One might conclude the cache line size in the L2 (or at least its behavior) is larger than the L1
line size or the L1 transaction size.
YANG ET AL.
7 of 12
8 of 12
YANG ET AL.
FIGURE 4
GPP hierarchical Roofline on the NVIDIA V100 GPU as a
function of stride and nw = 6
FIGURE 5
HPGMG GSRB hierarchical Roofline on the NVIDIA V100
GPU for the GSRB_FP implementation
4.2
HPGMG performance analysis
We evaluate HPGMG's three implementations of its GSRB smoother using the hierarchical Roofline model. As all variants are memory-intensive
(arithmetic intensity is always less than machine balance), unlike GPP, the FMA fraction of the instruction mix will play no role in our analysis.
Moreover, in lieu of an algorithmic parameter to affect changes in arithmetic intensity, in all of our analysis, we examine performance on each
level in the multigrid hierarchy HPGMG that runs on the GPU. Naively, one might think the same stencil should have the same intensity. However,
the deep ghost zone can reduce arithmetic intensity for the smaller boxes (lower levels), while cache effects can manifest and increase intensity
for the larger boxes (upper levels).
Figure 5 presents the hierarchical Roofline for the GSRB_FP variant of HPGMG's smoother as a function of level (level 5 with 8 163 boxes to
level 8 with 8 1283 boxes). Roofline clearly provides several immediate observations. First, performance is highly correlated with HBM bandwidth
(green line tracks the HBM ceiling). Second, HBM intensity increases with level. This should come as no surprise as intensity should scale as
O(dim3∕(dim + 4)3) as there is a ghost zone (two elements deep) on the high and low faces of each box. This substantially reduces intensity for
small boxes (dim = 16). Third, L1 intensity is roughly constant with box size. Once again, this should come as no surprise as, internally, the CUDA
implementation uses a fixed thread block dimension to tile each box. The constant thread block dimension exerts a constant pressure on the L1
cache. Fourth, there is substantial reuse in the L1 cache (L1(blue) and L2(red) intensities are widely separated), while there is virtually no reuse
in the L2 cache (L2(red) and HBM(blue) intensities are very close). This implies that, virtually, all reuse in HPGMG is captured by the L1 cache
or in register reuse and there is very little interthread block bandwidth filtering (something expected for tiled stencil computations). Finally, the
very astute will notice that the empirical HBM arithmetic intensity is twice the theoretical HPGMG intensity. This is an artifact of the GSRB_FP
implementation redundantly performing the stencil on every point and quashing the results by multiplying by the array of 1's and 0's—something
nvprof dutifully observes.
Figure 6 presents the hierarchical Roofline for the GSRB_BRANCH implementation. Recall that this implementation differs from the GSRB_FP
implementation in that it uses optimized modulo-2 arithmetic and a branch to avoid redundant computation and an extra (L1) load. As such, it
performs half as many (nonpredicated) floating-point operations and thus has half the performance and half the arithmetic intensity on each level
of multigrid and each level of the memory hierarchy as the GSRB_FP variant shown in Figure 5. All analysis and insights derived from GSRB_FP
apply to GSRB_BRANCH.
On paper, HPGMG's GSRB_STRIDE2 implementation seems like the ideal implementation. It performs no redundant work, all computation
remains converged/nonpredicated, and the stride-2 memory access pattern presented to the L1 should be filtered into a unit-stride pattern
presented to the L2/HBM. As such, it can be quite puzzling as to why the GSRB_STRIDE2 implementation underperforms the GSRB_BRANCH
and GSRB_FP variants. Our nvprof-based hierarchical Roofline model helps elucidate the causes.
8 of 12
YANG ET AL.
YANG ET AL.
9 of 12
FIGURE 6
HPGMG GSRB hierarchical Roofline on the NVIDIA V100
GPU for the GSRB_BRANCH implementation
FIGURE 7
HPGMG GSRB hierarchical Roofline on the NVIDIA V100
GPU for the GSRB_STRIDE2 implementation. Observe the
unexpected loss in L2 and HBM arithmetic intensity for the larger levels
Figure 7 shows the hierarchical Roofline for the GSRB_STRIDE2 variant. It should be immediately obvious that it looks quite different from
the GSRB_BRANCH version shown in Figure 6 with performance on the largest boxes (those that dominate HPGMG's solve time) substantially
lower. First, the trend in L1 intensity (x-coordinate) is very similar to GSRB_BRANCH. This indicates that as expected, each thread block accesses
memory in a similar manner to the GSRB_BRANCH variant. However, when looking at L2 and HBM intensity, we observe very different behaviors.
As one proceeds from level 5 (163) to level 6 (323), one observes increases in performance (ultimately bound by HBM bandwidth) but only
slight increases in arithmetic intensity. Conversely, from level 6 to level 7 (643) and level 8 (1283), we see substantial reductions in L2 and HBM
intensity. The former implies that unlike the GSRB_BRANCH variant, the GSRB_STRIDE2 variant is failing to capture locality in the L1 cache and
flooding L2 with additional data movement. The fact that HBM intensity is correlated with L2 intensity implies that the L2 is also failing to capture
any locality and transactions received by the L2 are passing through and becoming increased HBM data movement. Increasing data movement
when HBM-bound results in performance sliding down along the HBM Roofline.
4.3
TensorFlow conv2d performance analysis
In this section, we investigate the effects of different input parameters on the arithmetic intensity and performance of the TensorFlow conv2d
kernel. More precisely, we start with the baseline parameters described in Section 3.2 and then vary one parameter at a time. The parameters we
examine are batch size, number of output filters, and the kernel (or filter) size. The two data types, FP32 and FP16, are the precisions of the input
and output data but not necessarily those of the FLOPs or bytes measured, ie, TensorFlow may decide to execute in FP32 on FP16 inputs.
Batch size: Figure 8 depicts the impact of the batch size on performance (the batch size is the parameter N in Equation (5)). For FP32 kernels, ie,
when both input and output tensors are in FP32 type, the arithmetic intensity and GFLOP/s performance of conv2d do not change much simply
because the underlying algorithm is the same. There is an exception though, with the backward pass where TensorFlow has decided to call a
different wgrad subkernel from cuDNN for batch size 64, which has raised the performance a little bit compared with the other two batch sizes.
The FP16 kernels should follow the same trend, ie, same intensity and performance for all batch sizes. However, in the raw data, we observe a
significant increase in the data movement for larger batch sizes, and because we include housekeeping subkernels such as padding and shuffling
in our measurement, the performance of this fairly small kernel is severely affected by these essentially bandwidth-bound subkernels. In the next
experiment (number of output filters), where kernels are larger, this effect may be better amortized. However, for TensorFlow applications in
general, this could be one of the reasons why the peak performance of 125 TFLOP/s is very hard to reach.
The cuDNN library is called very frequently in TensorFlow, and it is observed in our raw data that cuDNN utilizes the shared memory on Volta
a lot. On the hierarchical Roofline charts, this is presented by the gaps between the L1 symbols and their respective L2 symbols, ie, these conv2d
kernels have good cache locality in level-one cache (including shared memory).
YANG ET AL.
9 of 12
10 of 12
YANG ET AL.
FIGURE 8
Effects of batch size on performance for forward (left panel) and backward pass (right panel) on second convolution layer from the
ResNet5023 network. Open symbols represent kernels with FP32 input and output, and filled symbols represent FP16
Number of output filters: Figure 9 shows the hierarchical Roofline for conv2d when the number of output filters increases, ie, C ′ in Equation (5).
The batch size for this set of results is fixed at 16. In both FP16 and FP32 cases and both forward and backward passes, the arithmetic intensity
and the GFLOP/s performance increase with the number of filters because the kernel becomes more compute intensive, ie, more computation is
done for the same amount of data movement. At the highest, the FP16 kernel in the backward pass is reaching 80% of the peak (at about 100
TFLOP/s), with 512 filters. In the meantime, the FP32 kernel gets even closer to the FMA(FP32) peak, at about 13 TFLOP/s. All of this shows
great promise of achieving Volta's compute capability with certain input parameters.
Kernel size: Figure 10 shows the hierarchical Roofline of conv2d as a function of the kernel size, ie, KH × KW, from 3 × 3, to 7 × 7, to 9 × 9. The
batch size here is fixed at 16, and the number of output filters is 64. The stride size is 2.
FIGURE 9
Effects of the number of output filters on performance for forward (left panel) and backward pass (right panel) on second convolution
layer from the ResNet5023 network. Open symbols represent kernels with FP32 input and output, and filled symbols represent FP16
FIGURE 10
Effects of kernel size on performance for forward (left panel) and backward pass (right panel) on second convolution layer from the
ResNet5023 network. Open symbols represent kernels with FP32 input and output, and filled symbols represent FP16
10 of 12
YANG ET AL.
YANG ET AL.
11 of 12
The increase in kernel size should have the same effect as the number of filters, ie, increased computation for the same amount of data
movement, hence increased arithmetic intensity and performance. However, there are two exceptions. One is the sudden drop in both arithmetic
intensity and performance at kernel size 9 × 9 in the forward pass for the FP32 input and output. In this case, the TensorFlow framework decides
to run a different set of subkernels instead of the wgrad subkernels for the 3 × 3 and 7 × 7 kernel sizes; it calls FFT subkernels underneath. This
could be due to mistakes in the autotuning decision-making process, where 9 × 9 seems large enough but it is still not at the level where FFT
could be run optimally. This phenomena is not observed in the backward pass case, which suggests there is certain sensitivity in TensorFlow's
autotuning mechanism to the heuristic data possibly from the warm-up stage.
The other exception is with the FP16 kernel in the backward pass at kernel size 9 × 9. Even though the input and output tensors are specified
to be FP16, the kernel is run in FP32 precision, ie, not on Tensor Cores. The input data is first converted to FP32, then the FP32 subkernels are
executed, and finally, the output is converted back to FP16. These unnecessary conversions lead to the overall kernel performance being even
worse than the natural FP32 kernel (with FP32 input and output, open triangles in Figure 10, right panel). It also suggests that the robustness of
the autotuning mechanism in TensorFlow could be improved.
5
SUMMARY, CONCLUSIONS, AND FUTURE WORK
In this paper, we extend the nvprof-based HBM Roofline methodology we developed for NVIDIA GPUs8 to capture the full NVIDIA GPU
memory hierarchy, the effects of FPADD/FPMUL in the instruction mix, the effects of reduced precision FP16 and FP32 Rooflines, and the
benefits of using FP16 Tensor Cores (HMMA instructions). To demonstrate the value of this hierarchical GPU Roofline methodology, we
used it to analyze three benchmarks: the moderately compute–intensive GPP Material Science proxy application, the cache-intensive HPGMG
AMR-multigrid proxy application, and the reduced precision and a Tensor Core-accelerated 2D convolution kernel from TensorFlow.
We observe that the hierarchical Roofline can capture insights into compute, cache, or memory performance bottlenecks as well as properties
of locality within each level of the cache hierarchy with performance being highly correlated with Roofline in the memory-intensive GPP and
HPGMG benchmarks. However, there were several cases in both HPGMG and TensorFlow where empirical performance and arithmetic intensity
diverged from Roofline or theoretical expectations. Similarly, the ultimate performance of GPP for high nw was only roughly correlated with the
‘‘partial’’ FMA performance ceiling derived from the FMA fraction of the instruction mix. We found that although Roofline provides observations
of performance metrics (eg, decreased arithmetic intensity), it does not inform users as to exactly what went wrong in their application's execution
or the code changes required to fix it. Nevertheless, it does provide some key first steps in potentially identifying areas of interest and may
motivate further experiments.
In the future, we will extend our Roofline methodology and usage along three axes, more ceilings, ie, more instruction-based Rooflines. As
for the first axis, we see several potential extensions to Roofline. In lieu of simply scaling performance or relying on marketing numbers, we
will extend ERT to support reduced precision (FP16 and FP32) and Tensor Cores (HMMA instructions). Moreover, whereas the FMA mix in the
instruction set was insufficient in determining the performance asymptote in GPP, we will extend our nvprof-based Roofline methodology to
incorporate occupancy in order to determine if we are expressing sufficiently thread-level parallelism to hide the GPU's high latencies. Moreover,
echoing our efforts to understand the impact of FPADD and FPMUL on FP64 performance, we will develop the requisite methodology to capture
and visualize the performance bottlenecks arising from mixed precision or Tensor Core–accelerated applications.
Although the traditional operation-oriented (GFLOP/s) Roofline model can readily assess performance and memory bottlenecks, it is poorly
suited for assessing either an application's exploitation of complex instruction set computing (CISC) or SIMD instruction set architectures (ISAs)
or the degree to which an application utilizes a processor's functional units, something that can manifest in mixed precision or partially vectorized
code. To that end, we will create an alternate Roofline methodology focused on floating-point instructions per cycle (IPCFP) and floating-point
instructions per byte. Recasting Roofline as such does not express performance (as it is agnostic of SIMD and FMA) but allows users to understand
when functional units can be fully utilized, executing a mix of reduced precision, Tensor Core, SIMD, or scalar instructions.
Finally, we will continue to apply our GPU Roofline methodology to more applications from a wider set of domains as well as extending our
methodology to other accelerated architectures.
ACKNOWLEDGMENTS
This material is based on work supported by the Advanced Scientific Computing Research Program in the U.S. Department of Energy, Office of
Science, under award number DE-AC02-05CH11231. This research used resources of the National Energy Research Scientific Computing Center
(NERSC), which is supported by the Office of Science of the U.S. Department of Energy under contract DE-AC02-05CH11231.
ORCID
Charlene Yang
https://orcid.org/0000-0002-0581-5845
Thorsten Kurth
https://orcid.org/0000-0003-0832-6198
YANG ET AL.
11 of 12
12 of 12
YANG ET AL.
REFERENCES
1.
Williams S, Waterman A, Patterson D. Roofline: an insightful visual performance model for multicore architectures. Commun ACM. 2009;52(4).
2.
Koskela T, Matveev Z, Yang C, et al. A novel multi-level integrated Roofline model approach for performance characterization. Paper presented at:
International Conference on High Performance Computing; 2018; Frankfurt, Germany.
3.
Williams S. Auto-Tuning Performance on Multicore Computers [PhD dissertation]. Berkeley, CA: University of California, Berkeley; 2008.
4.
NERSC LIKWID Documentation. https://www.nersc.gov/users/software/performance-and-debugging-tools-likwid/
5.
NERSC SDE Documentation. https://www.nersc.gov/users/application-performance/measuring-arithmetic-intensity/
6.
Barnes T, Cook B, Deslippe J, et al. Evaluating and optimizing the NERSC workload on Knights Landing. Paper presented at: 7th International
Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS); 2016; Salt Lake City, UT.
7.
Doerfler D, Deslippe J, Williams S, et al. Applying the Roofline performance model to the Intel Xeon Phi Knights Landing processor. Paper presented
at: International Conference on High Performance Computing; 2016; Frankfurt, Germany.
8.
Yang C, Gayatri R, Kurth T, et al. An empirical Roofline methodology for quantitatively assessing performance portability. Paper presented a: 2018
IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC; 2018; Dallas, TX.
9.
General Plasmon Pole (GPP) Kernel. https://github.com/cyanguwa/nersc-roofline
10.
HPGMG CUDA Code. https://bitbucket.org/nsakharnykh/hpgmg-cuda
11.
TensorFlow. https://tensorflow.org
12.
Empirical Roofline Toolkit (ERT). https://bitbucket.org/berkeleylab/cs-roofline-toolkit
13.
Python Matplotlib. https://matplotlib.org
14.
Example Scripts for Plotting Roofline. https://github.com/cyanguwa/nersc-roofline
15.
BerkeleyGW. https://berkeleygw.org
16.
Soininen J, Rehr J, Shirley EL. Electron self-energy calculation using a general multi-pole approximation. J Phys Condens Matter. 2003;15(17).
17.
HPGMG Website. https://hpgmg.org/
18.
HPGMG-FV Documentation. http://crd.lbl.gov/departments/computer-science/PAR/research/hpgmg
19.
AMReX Documentation. https://amrex-codes.github.io/amrex/
20.
Abadi M, Agarwal A, Barham P, et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. 2015. http://download.tensorflow.org/
paper/whitepaper2015.pdf
21.
tf.nn.conv2d Kernel. https://www.tensorflow.org/api_docs/python/tf/nn/conv2d
22.
Ben-Nun T, Hoefler T. Demystifying parallel and distributed deep learning: an in-depth concurrency analysis. ACM Comput Surv. 2018;52(4).
23.
He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. Paper presented at: IEEE Conference on Computer Vision and Pattern
Recognition (CVPR); 2015; Las Vegas, NV.
24.
PyCUDA Website. https://mathema.tician.de/software/pycuda
How to cite this article:
Yang C, Kurth T, Williams S. Hierarchical Roofline analysis for GPUs: Accelerating performance optimization
for the NERSC-9 Perlmutter system. Concurrency Computat Pract Exper. 2019;e5547. https://doi.org/10.1002/cpe.5547
How to cite this article: Yang C, Kurth T, Williams S. Hierarchical Roofline analysis for GPUs: Accelerating performance optimization for
the NERSC-9 Perlmutter system. Concurrency Computat Pract Exper. 2020;32:e5547. https://doi.org/10.1002/cpe.5547
12 of 12
YANG ET AL.


ORNL, August 2019
VOLTA TENSOR CORE TRAINING
2
AGENDA
V100 Architecture & Tensor Cores
Anatomy of a GEMM
Programming Approaches
Libraries
cublas
Iterative Refinement
Frameworks
WMMA & MMA.sync
CUTLASS
NVIDIA Tools
Case Studies
Asgard + HPL-AI
PICTC
DL Framework
Non-Traditional Uses
3
VOLTA
ARCHITECTURE AND
TENSOR CORES
4
TESLA V100
The Fastest and Most Productive GPU for Deep Learning and HPC
Volta Architecture
Most Productive GPU
Tensor Core
120 Programmable
TFLOPS Deep Learning
Improved SIMT Model
New Algorithms
Volta MPS
Inference Utilization
Improved NVLink &
HBM2
Efficient Bandwidth
5
21B transistors
815 mm2
80 SM
5120 CUDA Cores
640 Tensor Cores
16 GB HBM2
900 GB/s HBM2
300 GB/s NVLink
TESLA V100
*full GV100 chip contains 84 SMs
6
VOLTA GV100 SM
GV100
FP32 units
64
FP64 units
32
INT32 units
64
Tensor Cores
8
Register File
256 KB
Unified L1/Shared
memory
128 KB
Active Threads
2048
7
VOLTA TENSOR CORE
8
TENSOR CORE
Mixed Precision Matrix Math
4x4 matrices
D = AB + C
D =
FP16 or FP32
FP16
FP16
FP16 or FP32
A0,0
A0,1
A0,2
A0,3
A1,0
A1,1
A1,2
A1,3
A2,0
A2,1
A2,2
A2,3
A3,0
A3,1
A3,2
A3,3
B0,0
B0,1
B0,2
B0,3
B1,0
B1,1
B1,2
B1,3
B2,0
B2,1
B2,2
B2,3
B3,0
B3,1
B3,2
B3,3
C0,0
C0,1
C0,2
C0,3
C1,0
C1,1
C1,2
C1,3
C2,0
C2,1
C2,2
C2,3
C3,0
C3,1
C3,2
C3,3
9
VOLTA TENSOR OPERATION
FP16
storage/input
Full precision
product
Sum with
FP32
accumulator
Convert to
FP32 result
×
+
Also supports FP16 accumulator mode for inferencing
more products
F16
F32
F32
F16
10
TENSOR SYNCHRONIZATION
Warp-synchronizing operation
Full Warp 16x16 Matrix Math
Composed Matrix Multiply and
Accumulate for 16x16 matrices
Result distributed across warp
warp
11
FP32 AND FP16 REPRESENTATION
8-bit
exponent
23-bit
mantissa
10-bit
mantissa
5-bit
exponent
1.4 x 10-45 < x < 3.4 x 1038
5.96 x 10-8 < x < 65504
Dynamic Range
12
EFFICIENT LINEAR
ALGEBRA
COMPUTATIONS ON
GPUS
13
GENERAL MATRIX PRODUCT
Basic definition
General matrix product
C = α op(A) * op(B) + β C
C is M-by-N,  op(A) is M-by-K,  op(B) is K-by-N
Compute independent dot products
Inefficient due to large working sets to hold parts of A and B
// Independent dot products
for (int i = 0; i < M; ++i)
for (int j = 0; j < N; ++j)
for (int k = 0; k < K; ++k)
C[i][j] += A[i][k] * B[k][j];
14
GENERAL MATRIX PRODUCT
Accumulated outer products
General matrix product
C = α op(A) * op(B) + β C
C is M-by-N,  op(A) is M-by-K,  op(B) is K-by-N
Compute independent dot products
Permute loop nests
Load elements of A and B exactly once
// Independent dot products
for (int i = 0; i < M; ++i)
for (int j = 0; j < N; ++j)
for (int k = 0; k < K; ++k)
C[i][j] += A[i][k] * B[k][j];
// Accumulated outer products
for (int k = 0; k < K; ++k)
for (int i = 0; i < M; ++i)
for (int j = 0; j < N; ++j)
C[i][j] += A[i][k] * B[k][j];
15
GENERAL MATRIX PRODUCT
Computing matrix product one block at a time
Partition the loop nest into blocks along each dimension
• Partition into Mtile-by-Ntile independent matrix products
• Compute each product by accumulating Mtile-by-Ntile-by-Ktile matrix products
for (int mb = 0; mb < M; mb += Mtile)
for (int nb = 0; nb < N; nb += Ntile)
for (int kb = 0; kb < K; kb += Ktile)
{
// compute Mtile-by-Ntile-by-Ktile matrix product
for (int k = 0; k < Ktile; ++k)
for (int i = 0; i < Mtile; ++i)
for (int j = 0; j < Ntile; ++j)
{
int row = mb + i;
int col = nb + j;
C[row][col] +=
A[row][kb + k] * B[kb + k][col];
}
}
16
BLOCKED GEMM IN CUDA
Parallelism Among CUDA Thread Blocks
Launch a CUDA kernel grid
• Assign CUDA thread blocks to each partition of the output matrix
CUDA thread blocks compute Mtile-by-Ntile-by-K matrix product in parallel
• Iterate over K dimension in steps, performing an accumulated matrix product
for (int mb = 0; mb < M; mb += Mtile)
for (int nb = 0; nb < N; nb += Ntile)
for (int kb = 0; kb < K; kb += Ktile)
{
.. compute Mtile by Ntile by Ktile GEMM
}
by each CUDA thread block
17
THREAD BLOCK TILE STRUCTURE
Parallelism Within a CUDA Thread Block
Decompose thread block into warp-level tiles
• Load A and B operands into Shared Memory (reuse)
• C matrix distributed among warps
Each warp computes an independent matrix product
for (int kb = 0; kb < K; kb += Ktile)
{
.. load A and B tiles to shared memory
for (int m = 0; m < Mtile; m += warp_m)
for (int n = 0; n < Ntile; n += warp_n)
for (int k = 0; k < Ktile; k += warp_k)
.. compute warp_m by warp_n by warp_k GEMM
}
by each CUDA warp
18
WARP-LEVEL TILE STRUCTURE
Warp-level matrix product
Warps perform an accumulated matrix product
• Load A and B operands from SMEM into registers
• C matrix held in registers of participating threads
Shared Memory layout is K-strided for efficient loads
for (int k = 0; k < Ktile; k += warp_k)
{
.. load A tile from SMEM into registers
.. load B tile from SMEM into registers
for (int tm = 0; tm < warp_m; tm += thread_m)
for (int tn = 0; tn < warp_n; tn += thread_n)
for (int tk = 0; tk < warp_k; tk += thread_k)
.. compute thread_m by thread_n by thread_k GEMM
}
by each CUDA thread
19
THREAD-LEVEL TILE STRUCTURE
Parallelism within a thread
Threads compute accumulated matrix product
• A, B, and C held in registers
Opportunity for data reuse:
• O(M*N) computations on O(M+N) elements
for (int m = 0; m < thread_m; ++m)
for (int n = 0; n < thread_n; ++n)
for (int k = 0; k < thread_k; ++k)
C[m][n] += A[m][k] * B[n][k];
Fused multiply-accumulate instructions
20
COMPLETE GEMM HIERARCHY
Data reuse at each level of the memory hierarchy
21
TENSOR CORE
PROGRAMMING
MODELS
22
USING TENSOR CORES
Volta Optimized
Frameworks and Libraries
__device__ void tensor_op_16_16_16(
float *d, half *a, half *b, float *c)
{
wmma::fragment<matrix_a, …> Amat;
wmma::fragment<matrix_b, …> Bmat;
wmma::fragment<matrix_c, …> Cmat;
wmma::load_matrix_sync(Amat, a, 16);
wmma::load_matrix_sync(Bmat, b, 16);
wmma::fill_fragment(Cmat, 0.0f);
wmma::mma_sync(Cmat, Amat, Bmat, Cmat);
wmma::store_matrix_sync(d, Cmat, 16,
wmma::row_major);
}
CUDA C++
Warp-Level Matrix Operations
NVIDIA cuDNN, cuBLAS, TensorRT
23
CUBLAS TENSOR CORE HOW-TO
Math Mode set with cublasSetMathMode function.
Volta and Turing family Tensor Core can be used with in mixed precision (FP16 inputs, FP32 accumulation,
FP16 or FP32 output) routines.
Pure single precision routines use tensor core (when allowed) by down-converting inputs to half
(FP16) precision on the fly.
mathMode =
CUBLAS_DEFAULT_MATH
mathMode =
CUBLAS_TENSOR_OP_MATH
cublasHgemm, cublasSgemm,
cublasGemmEx(algo=DEFAULT)
Disallowed
Allowed
cublasGemmEx(algo=*_TENSOR_OP
Allowed
Allowed
Constraint: M,N,K,LDA,LDB,LDC and A,B,C pointers must ALL be aligned to 8
because of high memory bandwidth needed to efficiently use Tensor Cores.
24
CUBLAS FUTURE IMPROVEMENTS
●
Loosening constraints on Tensor Core usage:
1.
CUDA 10.1 Update 2 will lift some restrictions so that only requirements remaining are:
m%4 == 0
k%8 == 0
lda, ldb, ldc, A, B, C are aligned to 16 bytes,
2.
Plan to lift the restriction completely by adding new kernels to work on mis-aligned memory in a
future release.
●
Plans to make Tensor Core “opt-out” instead of “opt-in” for all directly applicable data type
combinations.
●
Plans to add NVTX based feedback to add information on tensor-core usage for detailed profiling.
Plans are subject to change
25
CUBLASLT: NEW MATRIX MULTIPLICATION LIBRARY
• Has its own header file, binary and lightweight
context
• Intended for power users of GEMMs that need
advanced features and optimizations for their
workflows
•
cuBLASLt is not a replacement for cuBLAS
• Adds flexibility in:
•
new matrix data layouts: IMMA, and planar complex
(Tensor Ops)
•
algorithmic implementation choices and heuristics
• Workspace support enables new optimizations –
e.g. split-k
• Non-traditional memory ordering enables hardware
optimizations such as INT8 IMMA on Turing GPUs
#include <cublasLt.h>
cublasLtCreate()
cublasLtMatmul()
cublasLtMatmulAlgoGetHeuristic()
cublasLtMatmulAlgoConfigSetAttribute()
solving linear system Ax = b
LU factorization
TENSOR CORE ACCELERATED IRS
SOLVING LINEAR SYSTEM AX = B
• LU factorization is used to solve a
linear system Ax=b
A  x = b
LUx = b
A
x
b
U
L
x
b
L
y
b
U
x
y
Ly    = b
then
Ux
= y
panel
update
step 1
step 2
step 3
step 4
nb
For s = 0, nb, .. N
1. panel factorize
2. update trailing matrix
GEMM
TRSM
Panel
L
U
LU factorization requires O(n3)
most of the operations are spent in GEMM
TENSOR CORE ACCELERATED IRS
SOLVING LINEAR SYSTEM AX = B
29
0
10
20
30
40
50
60
70
80
90
100
110
120
130
140
FP64
FP32
FP16 (TC)
V100 TFLOPS
TENSOR CORE ACCELERATED LIBRARIES
Multi-precision numerical methods
LU factorization used to solve Ax=b is
dominated by GEMMs
Can it be accelerated using Tensor Cores
and still get fp64 accuracy?
30
DIFFERENT LEVELS OF PRECISIONS USED DURING
FACTORIZATION WITH TENSOR CORES
FP32
FP32
FP16/FP32
FP32
FP32
FP16/FP32
…
m=n
2k 4k 6k 8k 10k 12k 14k 16k 18k 20k 22k 24k 26k 28k 30k
Tflop/s
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
FP16 TC square
FP16 TC k=256
FP16 square
FP16 k=256
FP32 square
FP32 k=256
FP64 square
FP64 k=256
STUDY OF THE MATRIX MATRIX
MULTIPLICATION KERNEL ON NVIDIA V100
•
dgemm achieve about 6.4 Tflop/s
•
sgemm achieve about 14   Tflop/s
•
hgemm achieve about 27   Tflop/s
•
Tensor cores gemm reach about 85 Tflop/s
•
Rank-k GEMM needed by LU does not
perform as well as square but still OK
Haidar et al., SC’18 proceedings
Results obtained using MAGMA 2.5.0 and GV100 GPU
matrix size
2k 4k 6k 8k 10k
14k
18k
22k
26k
30k
34k
Tflop/s
0
2
4
6
8
10
12
14
16
18
20
22
24
26
FP16-TC (Tensor Cores) hgetrf LU
FP16 hgetrf LU
FP32 sgetrf LU
FP64 dgetrf LU
• LU factorization is used to solve a
linear system Ax=b
A  x = b
LUx = b
Ly    = b
then
Ux
= y
Study of the LU factorization algorithm on Nvidia V100
LEVERAGING HALF PRECISION IN HPC ON V100
MOTIVATION
3~4X
A
x
b
U
L
x
b
L
y
b
U
x
y
Main idea is to use lower precision to compute the expensive flops (LU O(n3)) and then
iteratively refine the solution in order to achieve the FP64 arithmetic
•
Wilkinson, Moler, Stewart, & Higham provide error bound for SP fl pt results when using DP fl pt.
•
E. Carson and N. J. Higham. Accelerating the solution of linear systems by iterative refinement in three precisions.
•
It can be shown that using this approach we can compute the solution with residual similar to the 64-bit floating point precision.
Leveraging Tensor Cores
Iterative Refinement Solver
Flops = 2n3/(3 time)
meaning twice higher is twice faster
Problem generated with an arithmetic distribution of the singular values                                          and positive eigenvalues.
si = 1−( i−1
n−1)(1−
1
cond)
Tensor Core Accelerated IRS
solving linear system Ax = b Performance Behavior
•
solving Ax = b using FP64 LU
Flops = 2n3/(3 time)
meaning twice higher is twice faster
Problem generated with an arithmetic distribution of the singular values                                          and positive eigenvalues.
si = 1−( i−1
n−1)(1−
1
cond)
Tensor Core Accelerated IRS
solving linear system Ax = b Performance Behavior
•
solving Ax = b using FP64 LU
•
solving Ax = b using FP32 LU and
iterative refinement to achieve FP64
accuracy
Flops = 2n3/(3 time)
meaning twice higher is twice faster
Problem generated with an arithmetic distribution of the singular values                                          and positive eigenvalues.
si = 1−( i−1
n−1)(1−
1
cond)
Tensor Core Accelerated IRS
solving linear system Ax = b Performance Behavior
•
solving Ax = b using FP64 LU
•
solving Ax = b using FP32 LU and
iterative refinement to achieve FP64
accuracy
•
solving Ax = b using FP16 LU and
iterative refinement to achieve FP64
accuracy
4X
Flops = 2n3/(3 time)
meaning twice higher is twice faster
Problem generated with an arithmetic distribution of the singular values                                          and positive eigenvalues.
si = 1−( i−1
n−1)(1−
1
cond)
Tensor Core Accelerated IRS
solving linear system Ax = b Performance Behavior
•
solving Ax = b using FP64 LU
•
solving Ax = b using FP32 LU and
iterative refinement to achieve FP64
accuracy
•
solving Ax = b using FP16 LU and
iterative refinement to achieve FP64
accuracy
•
solving Ax = b using FP16 Tensor
Cores LU and iterative refinement to
achieve FP64 accuracy
Flops = 2n3/(3 time)
meaning twice higher is twice faster
Problem generated with an arithmetic distribution of the singular values                                          and positive eigenvalues.
si = 1−( i−1
n−1)(1−
1
cond)
Tensor Core Accelerated IRS
solving linear system Ax = b Performance Behavior
•
solving Ax = b using FP64 LU
•
solving Ax = b using FP32 LU and
iterative refinement to achieve FP64
accuracy
•
solving Ax = b using FP16 LU and
iterative refinement to achieve FP64
accuracy
•
solving Ax = b using FP16 Tensor
Cores LU and iterative refinement to
achieve FP64 accuracy
Flops = 2n3/(3 time)
meaning twice higher is twice faster
4X
Problem generated with an clustered distribution of the singular values
Tensor Core Accelerated IRS
solving linear system Ax = b Performance Behavior
•
solving Ax = b using FP64 LU
•
solving Ax = b using FP32 LU and
iterative refinement to achieve FP64
accuracy
•
solving Ax = b using FP16 LU and
iterative refinement to achieve FP64
accuracy
•
solving Ax = b using FP16 Tensor
Cores LU and iterative refinement to
achieve FP64 accuracy
Flops = 2n3/(3 time)
meaning twice higher is twice faster
Problem generated with an arithmetic distribution of the singular values                                         .
si = 1−( i−1
n−1)(1−
1
cond)
3X
Tensor Core Accelerated IRS
solving linear system Ax = b Performance Behavior
•
solving Ax = b using FP64 LU
•
solving Ax = b using FP32 LU and
iterative refinement to achieve FP64
accuracy
•
solving Ax = b using FP16 LU and
iterative refinement to achieve FP64
accuracy
•
solving Ax = b using FP16 Tensor
Cores LU and iterative refinement to
achieve FP64 accuracy
Convergence Checks
Is the solution really the same as the fp64 solver?
Leveraging Tensor Cores
Iterative Refinement Solver
Haidar et al., SC’18 proceedings
Results obtained using MAGMA 2.5.0 and GV100 GPU
43
Mixed-precision iterative refinement solver
GV100 vs TU102
Results obtained using MAGMA 2.5.0
0
2
4
6
8
10
12
14
16
18
20
22
0k
4k
8k
12k
16k
20k
24k
28k
32k
36k
40k
TFLOPS
Matrix Size
Performance of solving Ax= b to fp64 accuracy
(GV100 & TU102)
TU102 FP16-TC -> FP64
GV100 FP16-TC -> FP64
TU102 FP16 -> FP64
GV100 FP16 -> FP64
TU102 FP32 -> FP64
GV100 FP32 -> FP64
TU102 FP64
GV100 FP64
Mixed-precision iterative refinement solver
Energy Efficiency
Time (sec)
0
1
2
3
4
5
6
7
Average power CPU+GPU (Watts)
0
20
40
60
80
100
120
140
160
180
200
220
240
260
280
300
320
340
360
380
400
420
440
460
5.5
14
2021
Performance
in Tflop/s
Gflops/Watts
Joules
10.7
27
1041
16.8
48
609
24.0
74
470
Solving Ax=b on Nvidia V100
FP64 solver dgesv
FP32 --> 64 solver dsgesv
FP16 --> 64 solver dhgesv
FP16 --> 64 solver dhgesv (TC)
CPU: 10 cores E5-2650 v3
GPU: Nvidia V100
Haidar et al., SC’18 proceedings
Results obtained using MAGMA 2.5.0 and GV100 GPU
TENSOR CORE ACCELERATED ITERATIVE REFINEMENT
SOLVERS
• Real & Planar Complex
• FP32 & FP64 support
cuSOLVER productization plans
~September 2019
LU Solver
~November 2019
Cholesky
Solver
~November 2019
QR Solver
*Plans subject to change
47
AUTOMATIC MIXED PRECISION
Insert ~ two lines of code to introduce Automatic Mixed-Precision and get upto 3X speedup
AMP uses a graph optimization technique to determine FP16 and FP32 operations
Support for TensorFlow, PyTorch and MXNet
Easy to Use, Greater Performance and Boost in Productivity
Unleash the next generation AI performance and get faster to the market!
48
ENABLING AUTOMATIC MIXED PRECISION
Add Just A Few Lines of Code, Get Upto 3X Speedup
More details: https://developer.nvidia.com/automatic-mixed-precision
TensorFlo
w
os.environ['TF_ENABLE_AUTO_MIXED_PRECISION'] = '1'
OR
export TF_ENABLE_AUTO_MIXED_PRECISION=1
Explicit optimizer wrapper available in NVIDIA Container 19.07+, TF 1.14+, TF
2.0:
opt = tf.train.experimental.enable_mixed_precision_graph_rewrite(opt)
GA
PyTorch
model, optimizer = amp.initialize(model, optimizer, opt_level="O1")
with amp.scale_loss(loss, optimizer) as scaled_loss:
scaled_loss.backward()
GA
MXNet
amp.init()
amp.init_trainer(trainer)
with amp.scale_loss(loss, trainer) as scaled_loss:
autograd.backward(scaled_loss)
GA
Coming
Soon
49
ENABLING AUTOMATIC MIXED PRECISION
Add Just A Few Lines of Code
•
TensorFlow
•
NVIDIA container 19.03+:
• export TF_ENABLE_AUTO_MIXED_PRECISION=1 [automatic casting and automatic loss scaling]
•
Available in NVIDIA container 19.07+, TF 1.14+, TF 2.0:
•
We provide an explicit optimizer wrapper to perform loss scaling – which can also enable auto-casting for you:
import tensorflow as tf
opt = tf.train.GradientDescentOptimizer(0.5)
opt = tf.train.experimental.enable_mixed_precision_graph_rewrite(opt)
50
ENABLING AUTOMATIC MIXED PRECISION
Add Just A Few Lines of Code
•
PyTorch
•
Two steps: initialization and wrapping backpropagation
from apex import amp
model = …
optimizer = SomeOptimizer(model.parameters(), …)
# …
model, optimizer = amp.initialize(model, optimizer, opt_level=“O1”)
# …
for train_loop():
loss = loss_fn(model(x), y)
with amp.scale_loss(loss, optimizer) as scaled_loss:
scaled_loss.backward()
# Can manipulate the .grads if you’d like
optimizer.step()
51
ENABLING AUTOMATIC MIXED PRECISION
Add Just A Few Lines of Code
•
MXNET
•
NVIDIA container 19.03+ and MXNET 1.5:
from mxnet.contrib import amp
amp.init()
net = get_network()
trainer = mx.gluon.Trainer(...)
amp.init_trainer(trainer)
for data in dataloader:
with autograd.record(True):
out = net(data)
l = loss(out)
with amp.scale_loss(l, trainer) as scaled_loss:
autograd.backward(scaled_loss)
trainer.step()
52
AUTOMATIC MIXED PRECISION IN TENSORFLOW
Upto 3X Speedup
All models can be found at:
https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow, except for ssd-rn50-fpn-640, which is here: https://github.com/tensorflow/models/tree/master/research/object_detection
All performance collected on 1xV100-16GB, except bert-squadqa on 1xV100-32GB.
Speedup is the ratio of time to train for a fixed number of epochs in single-precision and Automatic Mixed Precision. Number of epochs for each model was matching the literature or common practice (it was also confirmed that both training sessions achieved the same model
accuracy).
Batch sizes:. rn50 (v1.5): 128 for FP32, 256 for AMP+XLA; ssd-rn50-fpn-640: 8 for FP32, 16 for AMP+XLA; NCF: 1M for FP32 and AMP+XLA; bert-squadqa: 4 for FP32, 10 for AMP+XLA; GNMT: 128 for FP32, 192 for AMP.
TensorFlow Medium Post: Automatic Mixed Precision in TensorFlow for Faster AI Training on NVIDIA GPUs
53
AUTOMATIC MIXED PRECISION IN PYTORCH
●
Plot shows ResNet-50 result with/without automatic mixed
precision(AMP)
●
More AMP enabled model scripts coming soon:
Mask-R CNN, GNMT, NCF, etc.
https://developer.nvidia.com/automatic-mixed-precision
FP32
AMP
Enabled
Mixed
Precision
Source: https://github.com/NVIDIA/apex/tree/master/examples/imagenet
2X
54
AUTOMATIC MIXED PRECISION IN MXNET
https://github.com/apache/incubator-mxnet/pull/14173
AMP speedup ~1.5X to 2X in comparison with FP32
(*) based on ResNet50 v1.5
55
CUDA TENSOR CORE PROGRAMMING
WMMA datatypes
wmma::fragment<matrix_a, …> Amat;
Per-Thread fragments to hold components of
matrices for use with Tensor Cores
56
CUDA TENSOR CORE PROGRAMMING
WMMA load and store operations
wmma::load_matrix_sync(Amat, a, stride);
Warp-level operation to fetch components of
matrices into fragments
warp
57
CUDA TENSOR CORE PROGRAMMING
WMMA Matrix Multiply and Accumulate Operation
wmma::mma_sync(Dmat, Amat, Bmat, Cmat);
Warp-level operation to perform matrix
multiply and accumulate
D =
58
CUDA TENSOR CORE PROGRAMMING
WMMA load and store operations
wmma::store_matrix_sync(d, Dmat, stride);
Warp-level operation to fetch components of
matrices into fragments
warp
Result
59
TENSOR CORE EXAMPLE
__device__ void tensor_op_16_16_16(
float *d, half *a, half *b, float *c)
{
wmma::fragment<matrix_a, …> Amat;
wmma::fragment<matrix_b, …> Bmat;
wmma::fragment<matrix_c, …> Cmat;
wmma::load_matrix_sync(Amat, a, 16);
wmma::load_matrix_sync(Bmat, b, 16);
wmma::fill_fragment(Cmat, 0.0f);
wmma::mma_sync(Cmat, Amat, Bmat, Cmat);
wmma::store_matrix_sync(d, Cmat, 16,
wmma::row_major);
}
CUDA C++
Warp-Level Matrix Operations
Create Fragments
Initialize Fragments
Perform MatMul
Store Results
60
TENSOR CORES IN CUDA FORTRAN
Similar to CUDA C WMMA API, with some name changes
real(2) support for half-precision data available (on both host and device) in PGI 19.7
compilers
Requires wmma Fortran module and macros in cuf_macros.CUF file
61
CUDA FORTRAN TENSOR CORE EXAMPLE
#include "cuf_macros.CUF"
module m
contains
attributes(global) subroutine wmma_16x16(a, b, c)
use wmma
real(2), intent(in) :: a(16,*), b(16,*)
real(4) :: c(16,*)
WMMASubMatrix(WMMAMatrixA, 16, 16, 16, Real, WMMAColMajor) :: sa
WMMASubMatrix(WMMAMatrixB, 16, 16, 16, Real, WMMAColMajor) :: sb
WMMASubMatrix(WMMAMatrixC, 16, 16, 16, Real, WMMAKind4) :: sc
sc = 0.0_4
call wmmaLoadMatrix(sa, a(1,1), 16)
call wmmaLoadMatrix(sb, b(1,1), 16)
call wmmaMatMul(sc, sa, sb, sc)
call wmmaStoreMatrix(c(1,1), sc, 16)
end subroutine wmma_16x16
end module m
Device Code
WMMA
Definitions
WMMA
“fragments”
Assignment
overloaded to call
fill_fragment()
62
CUDA FORTRAN TENSOR CORE EXAMPLE
program main
use m
use cudafor
integer, parameter :: m = 16, n=m, k=m
real(4) :: a(m,k), b(k,n), c(m,n), cref(m,n)
real(4), device :: c_d(m,n)
real(2), device :: ah_d(m,k), bh_d(k,n)
call random_number(a); a = int(4.*a); ah_d = a
call random_number(b); b = int(4.*b); bh_d = b
cref = matmul(a, b)
c = 0.0
call wmma_16x16<<<1,32>>>(ah_d, bh_d, c_d)
c = c_d
if (sum(abs(c-cref)) == 0.0) write(*,*) ‘Test passed’
end program main
Host Code
Launch with a single
warp of threads
Host-device
transfer and
4- to 2-byte
conversion
63
MMA.SYNC
64
mma.sync: new instruction in CUDA 10.1
• Directly targets Volta Tensor Cores
Matrix multiply-accumulate
D = A * B + C
• A, B: half
• C, D: float or half
Warp-synchronous:
• Four independent 8-by-8-by-4 matrix
multiply-accumulate operations
VOLTA MMA.SYNC
Warp-scoped matrix multiply instruction
65
VOLTA MMA.SYNC
Warp is partitioned into Quad Pairs
• QP0:    T0..T3     T16..T19
• QP1:    T4..T7     T20..T23
• QP2:    T8..T11    T24..T27
• QP3:   T12..T15    T28..T31
(eight threads each)
Each Quad Pair performs one 8-by-8-by-4
matrix multiply
Warp-scoped matrix multiply instruction
66
COMPOSING MATRIX MULTIPLIES
Replicate data to compute warp-wide 16-by-16-by-4 matrix product
• A0..7: QP0,QP2
A8..15: QP1, QP3
• B0..7: QP0,QP1
B8..15: QP2, QP3
1 x mma.sync: 16-by-16-by-4
67
VOLTA MMA.SYNC
D = A * B + C
PTX Syntax
mma.sync.aligned.m8n8k4.alayout.blayout.dtype.f16.f16.ctype
d, a, b, c;
.alayout = {.row, .col};
.blayout = {.row, .col};
.ctype
= {.f16, .f32};
.dtype
= {.f16, .f32};
d: 8 x .dtype
a: 4 x .f16
b: 4 x .f16
c: 8 x .ctype
Note: .f16 elements must be packed into .f16x2
https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-instructions-mma
68
THREAD-DATA MAPPING - F16 MULTIPLICANDS
Distributed among threads in quad pair (QP0 shown)
ROW-COL (“TN”)
COL-ROW (“NT”)
mma.sync.aligned.m8n8k4.alayout.blayout.dtype.f16.f16.ctype
d, a, b, c;
.alayout = {.row, .col};
.blayout = {.row, .col};
a: 2 x .f16x2
b: 2 x .f16x2
69
CUTLASS
CUDA C++ Template Library for Matrix Algebra
CUTLASS template library for GEMM computations
•
Blocked structure to maximize data reuse
•
Software pipelined to hide latency
•
Conflict-free Shared Memory access to maximize data throughput
See CUTLASS GTC 2018 talk.
70
CUTLASS DESIGN PATTERNS
Templates: generic programming and compile-time optimizations
Traits: describes properties, types, and functors used to specialize CUTLASS concepts
Params: structure containing parameters and precomputed values; passed to kernel as POD
Vectorized Memory Accesses: load and store as 32b, 64b, or 128b vectors
Shape<>: describes size of a 4D vector quantity
TileTraits<>: describes a 4D block of elements in memory
Fragment<>: partitioning of a tile across a collection of threads
TileIterator<>: loads a tile by a collection of threads; result is held in Fragment
Design patterns and template concepts in CUTLASS
71
GEMM TEMPLATE KERNEL
//
// CUTLASS GEMM kernel
//
template <typename Gemm>
__global__ void gemm_kernel(typename Gemm::Params params) {
// Declare shared memory
__shared__ typename Gemm::SharedStorage shared_storage;
// Construct the GEMM object with cleared accumulators
Gemm gemm(params);
// Compute the matrix multiply-accumulate
gemm.multiply_add(shared_storage.mainloop);
// Update output memory efficiently
gemm.update(shared_storage.epilogue);
}
//
// Specialization for single-precision
//
typedef cutlass::gemm::SgemmTraits<
cutlass::MatrixLayout::kColumnMajor,
cutlass::MatrixLayout::kRowMajor,
cutlass::Shape<8, 128, 128>
> SgemmTraits;
// Simplified kernel launch
Gemm<SgemmTraits>::launch(params);
CUTLASS provides building blocks for efficient device-side code
•
Helpers simplify common cases
72
EXAMPLE: VOLTA TENSOR CORES
WMMA: Warp-synchronous Matrix Multiply-Accumulate
•
API for issuing operations to Volta Tensor Cores
Targeting the CUDA WMMA API
/// Perform warp-level multiply-accumulate using WMMA API
template <
/// Data type of accumulator
typename ScalarC,
/// Shape of warp-level accumulator tile
typename WarpTile,
/// Shape of one WMMA operation – e.g. 16x16x16
typename WmmaTile
>
struct WmmaMultiplyAdd {
/// Compute number of WMMA operations
typedef typename ShapeDiv<WarpTile, WmmaTile>::Shape
Shape;
/// Multiply: D = A*B + C
inline __device__ void multiply_add(
FragmentA const & A,
FragmentB const & B,
FragmentC const & C,
FragmentD & D) {
// Perform M-by-N-by-K matrix product using WMMA
for (int n = 0; n < Shape::kH; ++n) {
for (int m = 0; m < Shape::kW; ++m) {
// WMMA API to invoke Tensor Cores
nvcuda::wmma::mma_sync(
D.elements[n][m],
A.elements[k][m],
B.elements[k][n],
C.elements[n][m]
);
}
}
}
};
73
CUTLASS 1.3
Reusable components targeting Volta Tensor Cores
GlobalLoadIterator
Transformer
SharedStoreIterator
SharedTileLoadIterator
MatrixMultiply
mma.sync
Transformer
SharedStoreIterator
SharedLoaditerator
GlobalLoadIterator
GlobalStoreIterator
Functor
GlobalLoadStream
Epilogue
Warp Matrix Multiply
74
STORING TO SHARED MEMORY
CUTLASS Tile Iterators to transform:
• Global Memory: Canonical matrix layout Shared Memory: permuted shared memory layout
cutlass/gemm/volta884_multiplicand.h
// Defines iterators for loading and storing multiplicands
template <
/// Identifies multiplicand of GEMM (A or B)
GemmOperand::Kind Operand,
/// Specifies layout of data in source memory
MatrixLayout::Kind Layout,
/// Specifies threadblock tile shape
typename Tile,
/// Specifies warp tile shape
typename WarpTile,
/// Specifies the number of participating warps
int WarpCount,
/// Specifies the delta between warp tiles
typename WarpDelta
>
struct Volta884Multiplicand {
//
// Thread-block load iterator (canonical matrix layout)
//
typedef ...  LoadIterator;
//
// Thread-block store iterator (permuted SMEM layout)
//
typedef ...  StoreIterator;
//
// Warp-level load iterator
//
typedef ... WarpLoadIterator;
};
75
LOADING FROM SHARED MEMORY
CUTLASS Tile Iterators to transform:
• Shared Memory: permuted shared memory layout Register File: mma.sync thread-data mapping
cutlass/gemm/volta884_multiplicand.h
// Defines iterators for loading and storing multiplicands
template <
/// Identifies multiplicand of GEMM (A or B)
GemmOperand::Kind Operand,
/// Specifies layout of data in source memory
MatrixLayout::Kind Layout,
/// Specifies threadblock tile shape
typename Tile,
/// Specifies warp tile shape
typename WarpTile,
/// Specifies the number of participating warps
int WarpCount,
/// Specifies the delta between warp tiles
typename WarpDelta
>
struct Volta884Multiplicand {
//
// Thread-block load iterator (canonical matrix layout)
//
typedef ...  LoadIterator;
//
// Thread-block store iterator (permuted SMEM layout)
//
typedef ...  StoreIterator;
//
// Warp-level load iterator
//
typedef ... WarpLoadIterator;
};
76
EXECUTING MMA.SYNC
CUTLASS Warp-scoped matrix multiply
• Register File: mma.sync thread-data mapping Tensor Cores: mma.sync
cutlass/gemm/volta884_multiply_add.h
template <
/// Shape of a warp-level GEMM (K-by-N-by-M)
typename WarpGemmShape_,
/// Layout of A multiplicand
MatrixLayout::Kind LayoutA,
/// Data type of A multiplicand
typename ScalarA,
/// Layout of B multiplicand
MatrixLayout::Kind LayoutB,
/// Data type of A multiplicand
typename ScalarB,
/// Data type of accumulators
typename ScalarC,
/// Whether infinite results are saturated to +-MAX_FLOAT
bool SatFinite = false
>
struct Volta884MultiplyAdd {
//
// Multiply : d = (-)a*b + c.
//
CUTLASS_DEVICE
void multiply_add(
FragmentA const& A,
FragmentB const& B,
Accumulators const& C,
Accumulators& D,
bool negate = false) {
...
}
};
77
SPEEDUP RELATIVE TO WMMA
1.06 1.10 1.10
1.25
1.37
1.41 1.42 1.43 1.43 1.44 1.44 1.45 1.45 1.45 1.46 1.46 1.46 1.46 1.47 1.47 1.50
1.61
1.66 1.67
1.71 1.71 1.73
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8transformer_1024_1280_4096_column_columntransformer_1024_1280_1024_column_columntransformer_1024_1280_1024_row columntransformer_1024_1280_33712_column_columntransformer_33712_5120_1024_row_columntransformer_1024_5120_33712_column_columntransformer_1024_1024_1280_column_rowtransformer_1024_1024_2560_column_rowtransformer_33712_1280_1024_row_columntransformer_33712_2560_1024_row_columntransformer_4096_5120_1024_row columntransformer_1024_5120_1024_column_columntransformer_4096_2560_1024_row columntransformer_1024_2560_1024_column_columntransformer_1024_1024_5120_column_rowtransformer_1024_5120_1024_row columntransformer_4096_1280_1024_row columntransformer_1024_2560_1024_row columntransformer_1024_2560_4096_column_columntransformer_1024_5120_4096_column_columntransformer_1024_2560_33712_column_columntransformer_1024_33712_5120_column_rowtransformer_1024_33712_2560_column_rowtransformer_1024_33712_1280_column_rowtransformer_1024_4096_5120_column_rowtransformer_1024_4096_2560_column_rowtransformer_1024_4096_1280_column_row
Speedup
Transformer - CUTLASS 1.3 - mma.sync speedup vs WMMA
V100 - CUDA 10.1
78
TENSOR CORES WITH VISUAL PROFILER
• Visual Profiler allows gathering of
Tensor Core Utilization after gathering
a timeline.
• Use the menu option “Run->Collect
Metrics and Events” to select the
“Tensor-Precision Function Unit
Utilization” metric under “Metrics-
>Multiprocessor”
79
TENSOR CORES WITH VISUAL PROFILER
After clicking on the kernel of
interest, select “GPU Details”
80
TENSOR CORES WITH NVPROF
• Nvprof supports the tensor_precision_fu_utilization metric which reveals the
utilization level of Tensor Cores in each kernel of your model. (Since CUDA9)
• The utilization level of the multiprocessor function units that execute tensor core
instructions on a scale of 0 to 10
Invocations
Metric Name
Metric Description
Min
Max
Avg
Device "Quadro GV100 (0)"
Kernel: compute_gemm(__half const *, __half const *, float const *, float*, float, float)
1
tensor_precision_fu_utilization
Tensor-Precision Function Unit Utilization
Mid (5)
Mid (5)
Mid (5)
nvprof -m tensor_precision_fu_utilization ./cudaTensorCoreGemm
81
TENSOR CORES WITH NSIGHT COMPUTE
• The Nsight Compute CLI allows collecting several metrics related to tensor
core usage
• This data can be view from the CLI or via the Nsight Compute GUI
nv-nsight-cu-cli --metrics sm__pipe_tensor_cycles_active.avg.pct_of_peak_sustained_active ./cudaTensorCoreGemm
[ compute_gemm, 2019-Aug-08 12:48:39, Context 1, Stream 7
Section: Command line profiler metrics
---------------------------------------------------------------------- --------------- ------------------------------
sm__pipe_tensor_cycles_active.avg.pct_of_peak_sustained_active
%
43.44
---------------------------------------------------------------------- --------------- ------------------------------
82
CASE STUDIES
83
ITERATIVE
REFINEMENT
(ASGARD & HPL)
With NVIDIA Tensor Cores the
simulations run 3.5X faster
than previous methods so the team
can simulate significantly longer
physical times and help advance our
understanding of how to sustain the
plasma and generate energy
ADVANCING FUSION DISCOVERIES
Joint work with ORNL & UTK David Green, Ed Azevedo, Wael Elwasif, Graham Lopez, Tyler McDaniel, Lin Mu, Stan Tomov, Jack Dongarra
ASGarD: Adaptive Sparse Grid Discretization
Two stream instability study
Scientists believe fusion is the future of energy but
maintaining plasma reactions is challenging and
disruptions can result in damage to the tokamak.
Researchers at ORNL are simulating instabilities in
the plasma to provide physicists a better
understanding of what happens
inside the reactor.
ADVANCING FUSION DISCOVERIES
Joint work with ORNL & UTK David Green, Ed Azevedo, Wael Elwasif, Graham Lopez, Tyler McDaniel, Lin Mu, Stan Tomov, Jack Dongarra
ASGarD: Adaptive Sparse Grid Discretization
Two stream instability study
magma_dgesv_gpu( N, nrhs,
d_A, ldda, ipiv,
d_B, lddb,
&info );
is replaced by
magma_dhgesv_iteref_gpu( N, nrhs,
d_A, ldda, h_ipiv, d_ipiv,
d_B, lddb, d_X, lddx,
d_workspace, &gesv_iter, &info);
Mixed-precision iterative refinement solver
Performance on a wider range of problems
PERFORMANCE FOR REAL-LIFE MATRICES FROM THE SUITESPARSE COLLECTION AND FROM DENSE MATRIX ARISING FROM RADAR DESIGN
name
Description
size
k• (A)
dgesv
dsgesv
dhgesv
dhgesv- TC
time(s)
# iter
time (s)
speedup
# iter
time (s)
speedup
# iter
time (s)
speedup
em192
radar design
26896
106
5.70
3
3.11
1.8328
40
5.21
1.0940
10
2.05
2.7805
appu
NASA app benchmark
14000
104
0.43
2
0.27
1.5926
7
0.24
1.7917
4
0.19
2.2632
ns3Da
3D Navier Stokes
20414
7.6 103
1.12
2
0.69
1.6232
6
0.54
2.0741
4
0.43
2.6047
nd6k
ND problem set
18000
3.5 102
0.81
2
0.45
1.8000
5
0.36
2.2500
3
0.30
2.7000
nd12k
ND problem set
36000
4.3 102
5.36
2
2.75
1.9491
5
1.86
2.8817
3
1.31
4.0916
Poisson
2D Poisson problem
32000
2.1 106
3.81
2
2.15
1.7721
59
2.04
1.8676
10
1.13
3.3717
Vlasov
2D Vlasov problem
22000
8.3 103
1.65
2
0.95
1.7368
4
0.67
2.4627
3
0.48
3.4375
appu
1,853,104
nnz
nd12k
14,220,946
nnz
nd12k
1,679,599
nnz
WORLD’S FASTEST SUPERCOMPUTER
TRIPLES ITS PERFORMANCE RECORD…
WORLD’S FASTEST SUPERCOMPUTER
TRIPLES ITS PERFORMANCE RECORD…
Using mixed precision iterative refinement approach we solved a matrix of order
10,091,520 on the DOE‘s Summit system.
Composed of nodes made up of 2 IBM Power-9 processors, 22 cores each, and 6 Nvidia V100
GPUs
The run used 4500 nodes
Used a random matrix with large diagonal elements to insure convergence of the method.
Mixed precision HPL achieved 445 PFLOPS
or 2.95X over DP precision HPL result on
the Top500. (148 PFLOPS)
89
TENSOR CORE FOR
PARTICLE PUSH IN MAGNETIC FIELD
90
PARTICLE PUSH
•
The governing equation for particle velocity in magnetic field is given
by:
•
ⅆ𝑣
ⅆ𝑡=
𝑞
𝑚v x B , 𝑣= 𝑣𝑒𝑙𝑜𝑐𝑖𝑡𝑦, 𝑞= 𝑐ℎ𝑎𝑟𝑔𝑒, 𝑚= 𝑚𝑎𝑠𝑠, 𝐵= 𝑚𝑎𝑔𝑛𝑒𝑡𝑖𝑐𝑓𝑖𝑒𝑙𝑑
•
*Discretizing the above equation in 2 dimension can lead to :
/*grab magnetic field at current position*/
B=EvalB(x);
/*get new velocity at n+1*/
v2[0] = v[0] + q/m*B*v[1]*dt;
v2[1] = v[1] - q/m*B*v[0]*dt;
/*update position*/
x2[0] = x[0] + v2[0]*dt;
x2[1] = x[1] + v2[1]*dt;
/*push down*/
v[0]=v2[0];
v[1]=v2[1];
(x1,y1)
(x2,y2)
(x3,y3)
(x4,y5)
Gather by magnetic forces from the cell
vertices.
*Ref: https://www.particleincell.com/2011/vxb-rotation/
91
BORIS METHOD
*Ref: https://www.particleincell.com/2011/vxb-rotation/
*Boris method is the de facto standard for particle pushing in plasma simulation codes
It is an explicit technique
The following equations summarize Boris method
𝑣+ −𝑣−
Δ𝑡
= 𝑞
2𝑚𝑣+ + 𝑣−× 𝐵
𝑣′ = 𝑣−+ 𝑣−× 𝑡
𝑣+ = 𝑣−+ 𝑣′ × s
𝑡= 𝑞
Τ
𝐵𝑚Δ Τ
𝑡2
𝑠=
2𝑡
1 + 𝑡2
In the absence of Electric Field. V+ acts as velocity update. Electric field can be easily added.
92
SCATTER PARTICLE INSTEAD OF GATHER
(x1,y1)
(x2,y2)
(x3,y3)
(x4,y5)
Gather by interpolation forces from the cell
vertices.
(x1,y1)
(x2,y2)
(x3,y3)
(x4,y5)
Scatter particle properties to nodes and add
compute at nodes.
To use Tensor core, scatter properties
of the particles and use WMMA to
compute and assemble
•
We separate velocity direction and magnitude. Magnitude in FP32 while directions in FP16
•
We pack velocity, t and s vectors into Tensor Core format. This is basically the scatter operation.
•
The GEMM updates velocities and add them back to particle final velocity at a given time step in FP32
93
IMPLEMENTING WITH WMMA
// Half-precision, no tensor core
int id = (int)(threadIdx.x % 32);
for(int k = 0;k<8;k++)
{
float t =0;
for(int i =0;i<16;i++)
{
t += __half2float(
__hmul(X[i + k*16] ,
Y[id + i * 32]) );
}
accmat[id + k*32] = t;
}
return;
// Half-precision, with tensor core
// Declare the fragments
nvcuda::wmma::fragment<nvcuda::wmma::matrix_a,
8, 32, 16, half, nvcuda::wmma::col_major>
a_frag;
nvcuda::wmma::fragment<nvcuda::wmma::matrix_b,
8, 32, 16, half, nvcuda::wmma::row_major>
b_frag;
nvcuda::wmma::fragment<nvcuda::wmma::accumulat
or, 8, 32, 16, float> acc_frag;
nvcuda::wmma::fill_fragment(acc_frag, 0.0f);
nvcuda::wmma::load_matrix_sync(a_frag, a, 8);
nvcuda::wmma::load_matrix_sync(b_frag, b, 32);
nvcuda::wmma::mma_sync(acc_frag, a_frag,
b_frag, acc_frag);
nvcuda::wmma::store_matrix_sync(c , acc_frag,
32, nvcuda::wmma::mem_row_major);
return;
94
PICTC PERFORMANCE COMPARISON
0
10
20
Reference
Tensor Cores
Reference
Tensor Cores
Time (ms) – Smaller is BEtter
Source: CUDA 10.1, Summit
2.1X
95
MACHINE LEARNING
96
DGX Mixed-Precision Led MLPerf
Time to Accuracy on Single Node
World’s Fastest Industry-Wide AI Benchmark Achieved on NVIDIA GPUs
Image Classification
RN50 v.1.5
MXNet
Object Detection
Mask R-CNN
PyTorch
Object Detection
SSD
PyTorch
Translation (recurrent)
GNMT
PyTorch
Translation (non-
recurrent)
Transformer
PyTorch
Recommendation
NCF
PyTorch
70 minutes
167 minutes
14 minutes
10 minutes
19 minutes
0.4 minutes
Test Platform: DGX-2H - Dual-Socket Xeon Platinum 8174, 1.5TB system RAM, 16 x 32 GB Tesla V100 SXM-3 GPUs connected via NVSwitch
97
97
NVIDIA NGC MODEL SCRIPTS
Tensor Core Optimized Deep Learning Examples
14 Available today!
●
Tensor Core optimized for greater performance
●
Test drive automatic mixed precision
●
Actively updated by NVIDIA
●
State-of-the-art accuracy using Tensor Cores
●
Serves as a reference implementation
●
Exposes hyperparameters and source code for further adjustment
Accessible via:
●
NVIDIA NGC https://ngc.nvidia.com/catalog/model-scripts
●
GitHub https://www.github.com/NVIDIA/deeplearningexamples
●
NVIDIA NGC Framework containers https://ngc.nvidia.com/catalog/containers
98
98
NVIDIA NGC MODEL SCRIPTS
Tensor Core Examples Built for Multiple Use Cases and Frameworks
A dedicated hub to download Tensor Core Optimized Deep Learning Examples on NGC
https://ngc.nvidia.com/catalog/model-scripts?quickFilter=deep-learning
99
99
MODEL SCRIPTS FOR VARIOUS APPLICATIONS
https://developer.nvidia.com/deep-learning-examples
Computer Vision
Speech & NLP
Recommender
Systems
●
SSD PyTorch
●
SSD TensorFlow
●
UNET-Industrial TensorFlow
●
UNET-Medical TensorFlow
●
ResNet-50 v1.5 MXNet
●
ResNet-50 PyTorch
●
ResNet-50 TensorFlow
●
Mask R-CNN PyTorch
●
GNMT v2 TensorFlow
●
GNMT v2 PyTorch
●
Transformer PyTorch
●
BERT (Pre-training and Q&A)
TensorFlow
●
NCF PyTorch
●
NCF TensorFlow
Text to Speech
●
Tacotron2 and WaveGlow
PyTorch
100
IMAGE CLASSIFICATION: MXNet ResNet-50 v1.5
https://ngc.nvidia.com/catalog/model-scripts/nvidia:resnet_50_v1_5_for_mxnet
NGC 18.12+ MXNet container
Source: https://github.com/NVIDIA/DeepLearningExamples/tree/master/MxNet/Classification/RN50v1.5
GPU:1xV100-16GB | DGX-1V | Batch Size: 208 (FP16), 96 (FP16)
DGX-1V 8GPU
16G
MXNet ResNet
FP32
MXNet ResNet
Mixed Precision
Time to Train
[Hours]
11.1
3.3
Train AccuracyTop
1%
76.67%
76.49%
Perf.
2,957
Img/sec
10,263
Img/sec
Data set
ImageNet
101
SPEECH SYNTHESIS: Tacotron 2 And WaveGlow v1.0
https://ngc.nvidia.com/catalog/model-scripts/nvidia:tacotron_2_and_waveglow_for_pytorch
Source: https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2
GPU:1xV100-16GB | DGX-1V | Batch Size: 208 (FP16), 96 (FP16)
DGX-1V
16G
Tacotron 2
FP32
Tacotron 2
Mixed
Precision
WaveGlow
FP32
WaveGlow
Mixed
Precision
Time to
Train
[Hours]
44
@ 1500
epochs
33.14
@ 1500
epochs
109.96
@ 1000
epochs
54.83
@ 1000
epochs
Train
Accuracy
Loss
(@1000
Epochs)
0.3629
0.3645
-6.1087
-6.0258
Perf.
10,843
tokens/sec
12,742
tokens/sec
257,687(*)
samples/sec
500,375(*)
samples/sec
Data set
LJ Speech Dataset
(*) With sampling rate equal to 22050, one second of audio is generated from 22050 samples
102
LANGUAGE MODELING: BERT for TensorFlow
https://ngc.nvidia.com/catalog/model-scripts/nvidia:bert_for_tensorflow
NGC 19.03 TensorFlow container
Source: https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT
GPU:8xV100-32GB | DGX-1 | Batch size per GPU: 4
DGX-1V 8GPU
32G
TF BERT
FP32
TF BERT
Mixed Precision
Time to Train
[Hours]
0.77
(BSxGPU = 4)
0.51
(BSxGPU = 4)
Train
F1 (mean)
90.83
90.99
Perf.
(BSxGPU = 4)
66.65
sentences/sec
129.16
sentences/sec
Data set
SQuaD (fine-tuning)
103
OBJECT DETECTION: TensorFlow SSD
NGC 19.03 TensorFlow container
https://ngc.nvidia.com/catalog/model-scripts/nvidia:ssd_for_tensorflow
Source: https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Detection/SSD
GPU:8xV100-16GB | DGX-1V | Batch Size: 32 (FP32, Mixed)
DGX-1V 8GPU
16G
TF SSD
FP32
TF SSD
Mixed Precision
Time to Train
1h 37min
1h 19min
Accuracy
(map)
0.268
0.269
Perf.
(BSxGPU = 32)
569
Img/sec
752
Img/sec
Data set
COCO 2017
104
TRANSLATION: PyTorch GNMT
https://ngc.nvidia.com/catalog/model-scripts/nvidia:gnmt_v2_for_pytorch
NGC 19.01 PyTorch container
Source: https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Translation/GNMT
GPU:16xV100-32GB | DGX-2 | Batch size: 128 (FP32, Mixed)
DGX-2V 16GPU
32G
PyTorch GNMT
FP32
PyTorch GNMT
Mixed Precision
Time to Train
[min]
58.6
26.3
Train Accuracy
BLEU score
24.16
24.22
Perf.
314.831
tokens/sec
738,521
tokens/sec
Data set
WMT16 English to German
105
RECOMMENDER: PyTorch Neural Collaborative Filter
https://ngc.nvidia.com/catalog/model-scripts/nvidia:ncf_for_pytorch
NGC 18.12 PyTorch container
Source: https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Recommendation/NCF
GPU:8xV100-16GB | DGX-1 | Batch size: 1,048,576
DGX-1V 8GPU
16G
PyTorch NCF
FP32
PyTorch NCF
Mixed Precision
Time to Accuracy
[seconds]
32.68
20.42
Accuracy
Hit Rate @10
0.96
0.96
Perf.
55,004,590
smp/sec
99,332,230
smp/sec
Data set
MovieLens 20M
106
INDUSTRIAL DEFECT DETECTION: TensorFlow U-Net
https://ngc.nvidia.com/catalog/model-scripts/nvidia:unet_industrial_for_tensorflow
NGC 19.03 TensorFlow container
Source: https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Segmentation/UNet_Industrial
GPU:8xV100-16GB | DGX-1 | Batch size: 16
DAGM 2007 has 10 classes (for the competition). Each class has an independent IOU.
DGX-1V 8GPU
16G
TF U-Net
FP32
TF U-Net
Mixed Precision
Time to Train
1 min 44 sec
1 min 36 sec
IOU
(Th=0.75 Class #4)
0.965
0.960
IOU
(Th=0.75 Class #9)
0.988
0.988
Perf.
445
Img/sec
491
Img/sec
Data set
DAGM 2007
107
Matching Accuracy for FP32 and Mixed Precision
Values are measured with model running on (1) DGX-1V 8GPU 16G, (2) DGX-1V 8GPU 32G or (3) DGX-2V 16GPU 32G
Model Script
Framework
Data Set
Automatic or
Manual Mixed-
Precision
FP32
Accuracy
Mixed-
Precision
Accuracy
FP32
Throughput
Mixed-Precision
Throughput
Speedup
BERT Q&A
(2)
TensorFlo
w
SQuaD
AMP
90.83
Top 1
90.99
Top 1
66.65
sentences/sec
129.16
sentences/sec
1.94
SSD w/RN50
(1)
TensorFlo
w
COCO 2017
AMP
0.268
mAP
0.269
mAP
569
images/sec
752
images/sec
1.32
GNMT
(3)
PyTorch
WMT16
English to
German
Manual
24.16
BLEU
24.22
BLEU
314,831
tokens/sec
738,521
tokens/sec
2.35
Neural
Collaborative
Filter
(1)
PyTorch
MovieLens
20M
Manual
0.959
HR
0.960
HR
55,004,590
samples/sec
99,332,230
items/sec
1.81
U-Net
Industrial
(1)
TensorFlo
w
DAGM 2007
AMP
0.965-
0.988
0.960-0.988
445
images/sec
491
images/sec
1.10
ResNet-50 v1.5
(1)
MXNet
ImageNet
Manual
76.67
Top 1%
76.49
Top 1%
2,957
images/sec
10,263
images/sec
3.47
Tacotron 2 /
WaveGlow 1.0
(1)
PyTorch
LJ Speech
Dataset
AMP
0.3629/
-6.1087
0.3645/
-6.0258
10,843 tok/s
257,687 smp/s
12,742 tok/s
500,375 smp/s
1.18/
1.94
108
NON-TRADITIONAL
USES
109
NON-TRADITIONAL USE OF TENSOR CORES
Many problems can be reformulated in terms of dense matrix multiplication
May not be most algorithmithically efficient, but tensor core performance can make up for
large constant factor differences
Example: computing correlations between sets of binary vectors
e.g. for clustering points in 0,1 𝑁[Joubert et al, 2018]
𝐶𝑖, 𝑗= ෍
𝑘
𝑁
𝐴𝑖𝑘∧𝐵𝑗𝑘
= ෍
𝑘
𝑁
𝐴𝑖𝑘
∗[𝐵𝑗𝑘]
⇒
𝐶𝑖𝑗= ෍
𝑘
𝐴𝑖𝑘𝐵𝑗𝑘
⇒
𝐶= 𝐴𝐵𝑇
Perfect use case for reduced precision: inputs are all in [0,1], outputs in [0,N] (or better)
See https://www.olcf.ornl.gov/wp-
content/uploads/2018/10/joubert_2019OLCFUserMeeting.pdf
When you have a GEMM-shaped hammer...
110
HGEMM VS GEMMEX
cublasSetMathMode(handle, CUBLAS_TENSOR_OP_MATH);
const __half *A = ...;
const __half *B = ...;
__half *C = ...;
cublasHgemm(handle, transa, transb, m, n, k,
alpha, A, lda, B, ldb,
beta, C, ldc);
...
float *C = ...;
cublasGemmEx(handle, transa, transb, m, n, k,
alpha, A, CUDA_R_16F, lda, B, CUDA_R_16F, ldb,
beta, C, CUDA_R_32F, ldc,
CUDA_R_32F,
CUBLAS_GEMM_DEFAULT_TENSOR_OP);
vs.
accumulates in FP16
exact results only up to N < 2048
accumulates in FP32
nearly as fast as cublasHgemm
(same datapath, just a bit more I/O)
exact results up to N < 224
make sure to ask for tensor cores!
111
PITFALL: LARGE VALUES FOR LD{A,B,C}
When N gets large, A and B matrices can
get very long and skinny
Prefer the memory layout that keeps lda
and ldb small
Change the transa/transb parameters on
cublas*Gemm* to match
Your caches and TLBs will thank you!
=
*
lda = 1000
ldb = 8000000
=
*
lda = 1000
ldb = 1000
(        )T


vs.
112
CONCLUSIONS
113
CONCLUSIONS
When used appropriately Tensor Cores can achieve as much as an 8X performance increase.
Real-world applications have seen > 2X performance improvement.
High-throughput Matrix Multiplication requires careful data considerations
A variety of High and Low-level approaches are available for programming Tensor Cores
Tensor Cores show promise beyond Machine Learning applications
114
ADDITIONAL RESOURCES
CUTLASS Basics - http://on-demand.gputechconf.com/gtc/2018/presentation/s8854-cutlass-
software-primitives-for-dense-linear-algebra-at-all-levels-and-scales-within-cuda.pdf
cuTensor & CUTLASS  -
https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/S9593/
cuBLAS - https://docs.nvidia.com/cuda/cublas/index.html
Mixed Precision Training Guide - https://docs.nvidia.com/deeplearning/sdk/mixed-precision-
training/index.html
CUDA C Programming Guide (WMMA) - https://docs.nvidia.com/cuda/cuda-c-programming-
guide/index.html#wmma
PTX ISA - https://docs.nvidia.com/cuda/parallel-thread-execution/index.html
CUDA Tensor Core Sample - https://docs.nvidia.com/cuda/cuda-samples/index.html#cuda-
tensor-core-gemm