ConvPIM: Evaluating Digital Processing-in-Memory through Convolutional
  Neural Network Acceleration
Processing-in-memory (PIM) architectures are emerging to reduce data movement
in data-intensive applications. These architectures seek to exploit the same
physical devices for both information storage and logic, thereby dwarfing the
required data transfer and utilizing the full internal memory bandwidth.
Whereas analog PIM utilizes the inherent connectivity of crossbar arrays for
approximate matrix-vector multiplication in the analog domain, digital PIM
architectures enable bitwise logic operations with massive parallelism across
columns of data within memory arrays. Several recent works have extended the
computational capabilities of digital PIM architectures towards the
full-precision (single-precision floating-point) acceleration of convolutional
neural networks (CNNs); yet, they lack a comprehensive comparison to GPUs. In
this paper, we examine the potential of digital PIM for CNN acceleration
through an updated quantitative comparison with GPUs, supplemented with an
analysis of the overall limitations of digital PIM. We begin by investigating
the different PIM architectures from a theoretical perspective to understand
the underlying performance limitations and improvements compared to
state-of-the-art hardware. We then uncover the tradeoffs between the different
strategies through a series of benchmarks ranging from memory-bound vectored
arithmetic to CNN acceleration. We conclude with insights into the general
performance of digital PIM architectures for different data-intensive
applications.
Orian Leitersdorf, Ronny Ronen, Shahar Kvatinsky
# ConvPIM: Evaluating Digital Processing-in-Memory through Convolutional Neural Network Acceleration

###### Abstract

Processing-in-memory (PIM) architectures are emerging to reduce data movement in data-intensive applications. These architectures seek to exploit the same physical devices for both information storage and logic, thereby dwarfing the required data transfer and utilizing the full internal memory bandwidth. Whereas analog PIM utilizes the inherent connectivity of crossbar arrays for _approximate_ matrix-vector multiplication in the analog domain, digital PIM architectures enable bitwise logic operations with massive parallelism across columns of data within memory arrays. Several recent works have extended the computational capabilities of _digital PIM_ architectures towards the _full-precision_ (single-precision floating-point) acceleration of convolutional neural networks (CNNs); yet, they lack a comprehensive comparison to GPUs. In this paper, we examine the potential of digital PIM for CNN acceleration through an updated quantitative comparison with GPUs, supplemented with an analysis of the overall limitations of digital PIM. We begin by investigating the different PIM architectures from a theoretical perspective to understand the underlying performance limitations and improvements compared to state-of-the-art hardware. We then uncover the tradeoffs between the different strategies through a series of benchmarks ranging from memory-bound vectord arithmetic to CNN acceleration. We conclude with insights into the general performance of digital PIM architectures for different data-intensive applications.

 Digital processing-in-memory (PIM), memory wall, convolutional neural networks (CNNs), floating-point numbers.

## 1 Introduction

The _memory wall_ serves as a fundamental bottleneck to computing systems for _memory-intensive_ applications as the memory access occasionally becomes orders of magnitude more expensive than the computation itself [1]. Therefore, processing-in-memory (PIM) solutions aim to tackle the memory wall via memory architectures with embedded processing capabilities. That is, the traditional read/write interface is supplemented with _logic_ operations that perform vectord computation within the memory without transferring the data through the memory wall bottleneck. While early proposals for PIM [2, 3] integrated small processors within the memory architecture, recent solutions perform logic by using the physical properties of the underlying memory devices themselves.

Numerous emerging PIM architectures exploit these physical properties towards _digital bitwise_ logic within memory arrays [4]. For example, memristive stateful logic [5, 6, 7, 8, 9] utilizes the memristor [10], an emerging physical device with variable resistance, to perform logic in the resistive domain on binary values (e.g., low resistance is logical one and high resistance is logical zero). For memristors connected as seen in Figure 1(a), applying fixed voltages at the memristor terminals causes the resistance of the output memristor to become conditional on the resistances of the input memristors (e.g., the logical NOR of the inputs). As this circuit appears within every row of a crossbar array of memristors (see Figure 1(b)), then we find that applying fixed voltages on the bitlines of the array can simultaneously induce a logic gate within every row of the crossbar array. Abstractly, we can consider a crossbar array as a binary matrix of memory, and stateful logic enables logic operations on columns of bits with \(O(1)\) time (e.g., NOR of two columns into a third column), see Figure 1(e). Furthermore, this abstract model covers additional architectures such as in-DRAM computing [11, 12]. This bitwise parallelism can be exploited towards high-throughput arithmetic (e.g., addition, multiplication) for both fixed-point and floating-point numbers within the memory in a bit-serial element-parallel fashion: the logic gates that construct the arithmetic function are performed serially, yet in parallel across all rows of an array for parallel vectorsd execution. This can provide massive throughout and energy benefits over traditional hardware such as GPUs _when the cache locality is low_ (e.g., vectored arithmetic on vectors stored only in the main memory) [13].

Recent works [14, 15, 16, 17] have proposed utilizing digital PIM architectures towards convolutional neural network (CNN) acceleration. While emerging _analog PIM_ approaches [18] provide significant acceleration over GPUs, they suffer from low accuracy due to noise in the analog domain and high costs for conversion between digital and analog domains. By supporting full-precision floating-point computation, digital

Fig. 1: Examples of digital PIM using (a, b) memristive [5, 6, 7] and (c, d) DRAM [11, 12] memories. Both follow (e) an abstract model of bitwise column operations in \(O(1)\) time. Figure adapted from [13].
PIM approaches have the potential to overcome these flaws for reliable CNN acceleration. FloatPIM [14] was the first such work - presenting vast improvement over GPU performance; several additional recent works [15, 16, 17] have since built upon the FloatPIM architecture and evaluation. Yet, there is a need for an updated comprehensive evaluation of these works and a comparison to state-of-the-art hardware as:

1. FloatPIM utilized several routines that were either erroneous (e.g., the floating-point addition algorithm only supported unsigned numbers [13, 15]) or have been since updated (e.g., convolution [19]).
2. The GPU baseline considered in FloatPIM (and later utilized in the additional works) stores the weights in the CPU memory during the computation [14]. In this paper, we demonstrate that storing the weights in the GPU memory significantly improves the GPU baseline.

This paper aims to simultaneously provide an updated evaluation of CNN acceleration with digital PIM while highlighting the underlying limitations of digital PIM architectures to give further insight into new applications. The paper is structured as follows. Section 2 presents the evaluation methodology of comparing memristive and DRAM PIM architectures to the NVIDIA A6000 GPU. The paper then progresses through several benchmarks in Sections 3, 4, and 5, first considering routines utilized in CNN acceleration and then various large-scale CNN models. We analyze the unique factors involved in each benchmark and develop several metrics that provide further insight into the performance of digital PIM architectures. Section 6 discusses the implications of these results, revealing the key characteristics that may indicate the potential of future applications for digital PIM acceleration.

## 2 Methodology

This section details the evaluation methodology1 that is utilized throughout this paper to compare the digital PIM architectures to GPUs, as summarized in Table I. While the exact parameters may vary depending on the specific GPU or PIM implementation, the overall trends evaluated in this paper remain.

Footnote 1: The code repository for this paper is publicly available at [https://github.com/deitersdorf/ConvPIM](https://github.com/deitersdorf/ConvPIM).

### _Gpu_

The GPU results provided throughout this paper are based on the NVIDIA A6000 GPU, including both experimental and theoretical results. The experimental results are derived from the PyTorch [20] library for general-purpose neural network acceleration, utilizing the built-in PyTorch profiler connected to the NVIDIA Nsight Systems profiler for GPU metrics (e.g., DRAM bandwidth, L1/L2 hit rate) and NVIDIA NVML [21] for power measurements. The theoretical results reflect _compute-bound_ performance and are thus derived from the theoretical peak computation throughput provided in the datasheet [22].

### _Digital PIM_

We consider a simple digital PIM architecture consisting of several crossbar arrays that may all operate simultaneously according to the abstract model depicted in Figure 1(e). We construct the architecture to match the overall GPU memory size of 48GB. The crossbar dimension, per-gate energy, and clock frequency are derived from state-of-the-art digital PIM architectures for memristive [23, 24] and DRAM [11, 12] PIM. The maximum power consumption is derived from the maximal parallelism at full duty cycle. We compare both throughput and normalized throughput per Watt (energy efficiency).

## 3 High-Throughput Arithmetic

We begin by analyzing in this section the elementary benchmark of memory-bound vectored arithmetic. Assume that two \(n\)-dimensional vectors \(\mathbf{u},\mathbf{v}\) of \(N\)-bit numbers (fixed-point or floating-point) reside in the main memory; the goal is to compute an element-wise elementary operation \(\circ\in\{+,-,*,/\}\) and store the result as an \(n\)-dimensional vector \(\mathbf{z}\) in the memory.

The bit-serial element-parallel approach extends the bitwise parallelism of digital PIM towards maximal arithmetic throughput. Consider an \(r\times c\) crossbar with two \(r\)-dimensional vectors \(\mathbf{u},\mathbf{v}\) stored with a single \(N\)-bit element per row (from each vector), as shown in Figure 2. This approach performs the arithmetic in parallel across all rows and all crossbars by constructing the arithmetic operation from a _serial_ sequence of logic gates. For example, \(N\)-bit fixed-point addition is performed by first constructing a 1-bit full-adder from 9 serial NOR gates [13, 25], and then performing ripple-carry addition by serially executing \(N\) full-adders. While the latency is high at \(9\cdot N=O(N)\) cycles, the throughput is also high at \(R/O(N)\) operations per cycle, where \(R\) is the total number of rows in the memory (i.e., \(r\) multiplied by the number of crossbars). Floating-point operations were originally considered incompatible with digital PIM [26] due to the control flow involved with the alignment and normalization of floating-point numbers. This hurdle was first overcome by FloatPIM [14]; however, several aspects of the design were erroneous [13, 15], and it also required a Content Addressable Memory (CAM) within every array. Conversely, AritPIM [13] recently proposed a suite of floating-point algorithms that adheres to the IEEE 754 standard exactly while requiring no modifications to the abstract model.

We compare the performance of the suite of arithmetic functions proposed in AritPIM [13] to experimental and theoretical GPU performance in Figure 3. The experimental GPU performance is bounded by the memory bandwidth for reading the input vectors \(\mathbf{u},\mathbf{v}\) and writing the result \(\mathbf{z}\), as indicated by the \(>94\%\) DRAM memory bandwidth recorded across all functions. Therefore, it depends only on the bit width of the underlying arithmetic operation. The theoretical compute-bound

\begin{table}
\begin{tabular}{|c|l|} \hline
**Configuration** & **Parameters** \\ \hline \hline \multirow{6}{*}{NVIDIA A6000 GPU} & _Number of Cores:_ 10752 \\  & _Memory Size:_ 48 GB \\  & _Memory Bandwidth:_ 768 GB/s \\  & _Clock Frequency:_ 1410 MHz \\  & _Max Power:_ 300W \\ \hline \multirow{6}{*}{Memristive/DRAM PIM} & _Crossbar Size:_ 1024 / 65536 \(\times\) 1024 \\  & _Memory Size:_ 48 GB \\ \cline{1-1}  & _Gate Energy:_ 6.4 f / 391 f \\ \cline{1-1}  & _Clock Frequency:_ 333 MHz / 0.5 MHz \\ \cline{1-1}  & _Max Power:_ 860 W / 80 W \\ \hline \end{tabular}
\end{table} TABLE I: Evaluation Parameters

Fig. 2: The bit-serial element-parallel approach to high-throughput in-memory arithmetic that performs vectored operations in an \(r\times c\) crossbar as a sequence of parallel logic operations on columns [13].
GPU performance reflects the theoretical throughput provided in an ideal circumstance where memory operations are not required. For digital PIM, in the spirit of [27], we define the _compute complexity_ (CC) as the number of logic gates performed per bit (e.g., \(9N/(3N)=3\) for \(N\)-bit fixed-point addition as the inputs and outputs occupy \(3N\) bits). Naturally, we find an inverse relationship between the CC of an arithmetic operation and the improvement over experimental GPU performance, as shown in Figure 4. Therefore, PIM is most effective compared to memory-bound GPU when the CC is low. Notice that 16-bit and 32-bit addition possess the same CC as PIM addition latency is linear in \(N\) (doubling \(N\) doubles the PIM latency).

## 4 Matrix Multiplication and Convolution

This section investigates the performance of matrix multiplication and 2D convolution via digital PIM. The extension of the arithmetic parallelism provided in the previous section to such matrix operations was first investigated in FloatPIM [14] and then generalized in MatPIM [19]. These works express the matrix operations as a serial sequence of vectored arithmetic operations, thereby utilizing digital PIM for the vector parallelism. The matrix operations are characterized by high data reuse, and thus the experimental memory-bound GPU approaches the theoretical compute-bound GPU performance.

Figure 5 compares the performance of digital PIM approaches to experimental and theoretical GPU performance for _batched_ matrix multiplication on many pairs of matrices (both of dimension \(n\times n\)). In this operation, we find reuse of \(O(n)\) as there is a total of \(O(n^{3})\) operations operating on \(O(n^{2})\) data. Therefore, as \(n\) increases, we expect the performance gap between experimental memory-bound GPU and theoretical compute-bound GPU will diminish. Indeed, we find in Figure 5 that the gap shrinks with increasing \(n\) (e.g., \(n=32\) has a significantly larger gap than \(n=128\)). Overall, we conclude that starting at \(n=128\), the GPU performance surpasses that of digital PIM due to the data reuse mitigating the memory wall bottleneck. Two-dimensional convolution with a \(k\times k\) kernel on a \(W\times H\) image possesses similar considerations with data reuse of \(O(k^{2})\) (\(O(WHk^{2})\) operations on \(O(WH)\) data).

## 5 CNN Inference and Training

We culminate the benchmark analysis by evaluating full-precision inference and training of convolutional neural networks (CNN). The acceleration of such neural networks primarily involves matrix multiplication for fully connected layers and 2D convolution for the convolutional layers, as well as element-wise operations for activation functions (e.g., ReLU). We consider a benchmark consisting of the AlexNet [28], GoogLeNet [29] and ResNet-50 [30] convolutional neural networks with the ImageNet [31] dataset. We evaluate the digital PIM performance by considering only the required matrix multiplication and 2D convolution operations, thereby providing an upper bound on the digital PIM performance that is also close to the true performance. The experimental GPU performance is measured through the PyTorch [20] implementations of the CNNs with random input images of size \(224\times 224\times 3\)[31].

In Figure 6, we compare the upper-bound digital PIM performance with the GPU performance on CNN inference.

Fig. 4: Inverse relationship between compute complexity and digital PIM improvement over memory-bound GPU (experimental).

Fig. 5: Comparison of throughput (matrix multiplications per second) and normalized throughput per Watt (energy efficiency) for matrix multiplication with \(n\times n\) matrices of 32-bit floating-point numbers.

Fig. 3: Comparison of throughput (operations per second) and normalized throughput per Watt (energy efficiency) for the addition and multiplication of 32-bit fixed-point and 32-bit floating-point (FP) numbers.
We find that the experimental GPU performance is close to the theoretical peak performance across all models due to the moderately-high data reuse (\(55-67\%\) L2 hit rate) - notice that the gap in ResNet and GoogLeNet is more significant than AlexNet since some of their operations are with low reuse (e.g., residual connections, \(1\times 1\) convolutions). For the same reason, we find that the digital PIM performance is not significantly better than the GPU performance, and digital PIM energy is slightly worse. The results for CNN training are provided in the code repository and portray similar trends to Figure 6.

## 6 Discussion and Conclusion

Recent works have proposed to utilize emerging digital bit-wise PIM approaches towards the full-precision acceleration of CNNs as an alternative to analog PIM. This approach benefits from the high accuracy of digital floating point operations and the reduction in data transfer due to the in-memory operations. However, through a series of benchmarks starting with element-wise vector operations and culminating in CNN inference and training, we demonstrate that digital PIM approaches, with current parameters, are unable to surpass GPU performance for full-precision CNN acceleration.

We identify the poor performance of digital PIM in full-precision CNN acceleration as arising from a combination of two factors: high compute complexity for the underlying arithmetic operations, and high data reuse in CNN architectures. The analysis in Figure 4 reveals that floating-point multiplication possesses relatively high compute complexity and thus already has a low throughput improvement over GPU performance. Furthermore, Figure 3 demonstrates that this improvement originates entirely from the memory wall bottleneck throttling the GPU performance as the theoretical compute-bound GPU results surpass digital PIM. Therefore, when we increase the data reuse in Figure 5, we find that the PIM performance becomes inferior to GPU performance as the memory wall is no longer the bottleneck. That is, we find that it is the combination of the high compute complexity and the high data reuse that leads to the inferior digital PIM performance in CNN acceleration. Hence, future work may focus on applications that prioritize arithmetic operations with low compute complexity or low data reuse in GPUs.

## Acknowledgments

This work was supported by the European Research Council through the European Union's Horizon 2020 Research and Innovation Programme under Grant 757259, by the European Research Council through the European Union's Horizon Research and Innovation Programme under Grant 101069336, and by the Israel Science Foundation under Grant 1514/17.
