NeuralMatrix: Compute the Entire Neural Networks with Linear Matrix
  Operations for Efficient Inference
The inherent diversity of computation types within the deep neural network
(DNN) models often requires a variety of specialized units in hardware
processors, which limits computational efficiency, increasing both inference
latency and power consumption, especially when the hardware processor needs to
support and execute different neural networks. In this study, we introduce
NeuralMatrix, which elastically transforms the computations of entire DNNs into
linear matrix operations. This transformation allows seamless execution of
various DNN models all with matrix operations and paves the way for running
versatile DNN models with a single General Matrix Multiplication (GEMM)
accelerator.Extensive experiments with both CNN and transformer-based models
demonstrate the potential of NeuralMatrix to accurately and efficiently execute
a wide range of DNN models, achieving 2.17-38.72 times computation efficiency
(i.e., throughput per power) compared to CPUs, GPUs, and SoC platforms. This
level of efficiency is usually only attainable with the accelerator designed
for a specific neural network.
Ruiqi Sun, Siwei Ye, Jie Zhao, Xin He, Jianzhe Lin, Yiran Li, An Zou
NeuralMatrix: Moving Entire Neural Networks to General Matrix Multiplication for Efficient Inference

###### Abstract

In this study, we introduce NeuralMatrix, a novel framework that enables the computation of versatile deep neural networks (DNNs) on a single general matrix multiplication (GEMM) accelerator. The proposed approach overcomes the specificity limitations of ASIC-based accelerators while achieving application-specific acceleration levels compared to general-purpose processors such as CPUs and GPUs. We address the challenges of mapping both linear and nonlinear operations in DNN computation to general matrix multiplications and the impact of using a GEMM accelerator on DNN inference accuracy. Extensive experiments are conducted on various DNN models from three popular categories (i.e., CNN, Transformers, and GNN) as illustrative backbone models. Our results demonstrate that DNNs suffer only up to a 2.02% accuracy loss after being converted to general matrix multiplication, while achieving 113x to 19.44x improvements in throughput per power compared to CPUs and GPUs.

## 1 Introduction

In recent years, deep learning networks (DNNs) have found applications in a wide range of scenarios, leading to the development of various types of neural networks. As neural network architectures continue to expand in size and complexity, they pose substantial computational challenges for various platforms. Application-specific integrated circuits (ASICs) offer a potential solution for supporting DNNs on mobile and edge devices. For example, Bai et al. [1] introduced a CNN accelerator design that incorporates a multiplier array, add tree, normalization, ReLU, and pooling units. Similarly, Thierry Tambe et al. [2] proposed an edge transformer accelerator featuring processing units (with floating-point vector and accumulate) and dedicated function units for layer normalization, softmax, and other unique operators in each layer.

ASIC-based accelerators are known for their efficient execution of specific DNN applications. However, their inherent specificity, including the types and numbers of functional units, can restrict their adaptability. For example, transformer-based BERT uses 72.5% of its computation cycles for versatile nonlinear operations [2], necessitating the integration of specific types and amounts of nonlinear functional units in its accelerator. However, when the same accelerator is used for other networks, such as CNN and GNN, which have far fewer nonlinear operations, these functional units can become unnecessary burdens. Consequently, a significant gap exists between the generality and computation efficiency of the accelerator when it runs versatile DNN applications [3].

In this study, we introduce NeuralMatrix, a framework that combines the best of both worlds, as illustrated in Fig. 1. On one hand, it overcomes the specificity limitations of ASIC-based accelerators and enables the computation of versatile DNNs on a single general matrix multiplication (GEMM) accelerator. On the other hand, compared to other general-purpose processors such as CPUs and GPUs, NeuralMatrix achieves application-specific acceleration levels by converting DNNs to general matrix multiplication and executing them with GEMM accelerators. Supporting different DNN architectures on a single GEMM accelerator is not trivial. Several challenges need to be addressed, including how to map both linear and nonlinear operations in DNN computation to general matrix
ultiplications so that a GEMM can fully support them, and the impact of using a GEMM accelerator on DNN inference accuracy. To address the first challenge, NeuralMatrix employs a fundamental GEMM mapping layer through which linear operations in the networks are reshaped and reorganized for matrix multiplications. Specifically, NeuralMatrix adopts a novel approach to circumvent the GEMM accelerator's limitation in dealing with nonlinear operations: they are approximated using piecewise functions if needed and further mapped to matrix multiplication through a segmented approach. To address the second challenge, we propose two NeuralMatrix variants that differentiate based on whether the approximation is conducted pre- or post-finetuning, and present extensive experiments to compare the two approaches. To the best of our knowledge, we are the first to transform entire DNNs into general matrix multiplication and execute them using a single GEMM accelerator. The proposed framework, NeuralMatrix, naturally handles the irregularity across versatile DNNs, achieving both generality and computational efficiency.

We conducted extensive experiments to verify the feasibility of NeuralMatrix by testing various DNN models from three popular categories (i.e., CNN, transformer, and GNN) as illustrative backbone models. Our results indicate that DNNs suffer only up to a 2.02% accuracy loss after being converted to general matrix multiplication. Both floating-point computation and integer quantization are included in each network model. Compared to CPUs and GPUs, NeuralMatrix can achieve 113x to 19.44x improvements in throughput per power. In comparison with classic ASIC designs, the accuracy loss is negligible while achieving similar throughput per power (i.e., 86.0% - 310.6%).

## 2 Background and Related Work

### General Matrix Multiplication Accelerator

General Matrix Multiplication (GEMM) accelerators are specialized hardware components or processors developed to expedite matrix multiplication operations [4]. They are widely employed in data centers for high-performance computing and in edge devices to boost the efficiency of various tasks, such as digital signal processing, artificial intelligence, and scientific simulations [5]. GEMM accelerators can be integrated with dedicated devices such as Tensor Processing Units (TPUs) [6], incorporated into System-on-Chip (SoC) configurations [7], or designed as standalone chips [8]. In this study, we specifically focus on GEMM accelerators as standalone hardware processors for mobile device applications, aiming to improve computational efficiency on low-resource hardware.

In a nutshell, GEMM comprises numerous parallel processing elements (PEs) and hierarchical memory (i.e., L1, L2, and L3), as shown in Fig. 1. Within each PE, the multiply-and-accumulate (MAC) unit performs computations, while the local (L1) buffer stores the MAC unit's inputs and partial sums. Multiple PEs are arranged in an array to enable parallel execution of many MAC operations. The network-on-chip (NoC) facilitates data transmission between the local (L1) buffers inside each PE and the global (L2) buffer for GEMM. Additionally, a high-bandwidth off-chip (L3) memory serves as a backup for providing input data and holding output data. Because data access energy and latency increase linearly from L1 to L2 and become orders of magnitude larger in L3 [4], GEMM accelerators are designed to maximize data reuse within on-chip L1 and L2 buffers.

Compared to CPUs and GPUs, which accommodate a variety of instructions through a range of logic and arithmetic components, GEMM accelerators are explicitly designed for matrix multiplication using only multiply-and-accumulate units and buffers. This focused approach to matrix multiplication results in exceptional efficiency [9]. As a result, employing the entire neural network on a single GEMM accelerator can lead to considerable efficiency gains, which is the central focus of this work.

Figure 1: NeuralMatrix translates neural network computation tasks into general matrix multiplication, enabling them to be fully supported by general matrix multiplication (GEMM) accelerators.
### Related Work

The growing size and computational demands of DNNs pose significant challenges. To tackle these issues, various techniques have been proposed, including network pruning [10], compression [2], quantization [11], and early exit [12], all of which aim to minimize network size. On the hardware side, graphics processing units (GPUs) and tensor processing units (TPUs) [13] are employed as general-purpose processors to speed up neural networks. However, these processors incorporate numerous processing units to manage diverse computations, resulting in wasted chip area, power, and energy when only some processing units are utilized, and others remain idle [13].

In pursuit of enhanced efficiency, researchers have developed application-specific integrated circuits (ASICs) to accelerate particular neural networks [14; 15; 16; 17; 18]. Data movement can often become the bottleneck, but ASICs can alleviate this by integrating memory and other processing units on the same chip, reducing the need for data transfer between components [19]. Moreover, ASICs are tailored to execute specific algorithms or sets of algorithms by incorporating only the required functional components for the tasks. Consequently, ASICs offer lower power consumption and improved performance compared to general-purpose processors. However, despite their success in accelerating DNNs, ASICs' specialized features restrict their versatility in boosting diverse applications [3].

The GEMM unit, a processing unit found in Google's TPUs and many ASIC accelerators, is designed to speed up matrix multiplication and accumulation. For instance, Zhou et al. [20] characterized the implicit convolution algorithm on commercial matrix-multiplication accelerators, while Vasudevan et al. [21] accelerated parallel multi-channel convolution using general matrix multiplication. Although the GEMM unit can accelerate matrix operations and general parallel linear operations, it cannot handle nonlinear operations.

To overcome this limitation, we propose NeuralMatrix, a solution that fully utilizes general matrix multiplication (GEMM) to accelerate classic DNNs. By leveraging standalone GEMM units in accelerators, NeuralMatrix enables the acceleration of various neural networks using general-purpose GEMM accelerators. Unlike existing approaches that categorize and accelerate different types of irregularities in neural networks, as surveyed in [3], NeuralMatrix simplifies all computations, including both linear and nonlinear operations, to a single GEMM operation. This approach eliminates the need for additional functional units and data transfers between units, allowing NeuralMatrix to achieve application-specific efficiency using general-purpose GEMM accelerators.

Figure 2: Overview of NeuralMatrix. Different types of DNN computation will go through different decision and process steps. Eventually, an entire neural network can be moved to matrix multiplication and become fully supported by a GEMM accelerator.
NeuralMatrix - Running Networks with Matrix Multiplication

Although GEMM accelerators exhibit remarkable performance and efficiency in carrying out matrix multiplication and addition operations, they are inadequate on their own for handling the computation of various DNNs. In this section, we describe how NeuralMatrix addresses this constraint. Given a computation need from a DNN, NeuralMatrix runs a series of decision and computation processes. Its high-level logic is depicted by the flow-chart in Fig. 2. First, the computation in neural networks can be classified into linear and nonlinear operations. Linear operations are directly mapped to GEMM accelerators through GEMM mapping (SS 3.1). Among nonlinear operations, NeuralMatrix will then decide if one operation already corresponds to a piecewise linear function (e.g., ReLU), which can be computed using the piecewise linear calculation method. If not, an offline piecewise approximation will be performed before it can be handled by piecewise linear calculations (SS 3.2). Considering the relation between the aforementioned approximation and finetuning, we introduce two variations of NeuralMatrix, specifically the _post-approximation_ and _pre-approximation_ methods, and discuss their potential impact on the final inference accuracy (SS 3.3).

### GEMM Mapping

Linear operations are pervasive in DNNs, for example in fully connected layers, convolution kernels, attention mechanisms, and more. These linear operations involve 2D, 3D, or higher-dimensional tensors. We first introduce our GEMM mapping module, which is the foundation of NeuralMatrix since it is the interface to the GEMM accelerators. By applying reshaping and re-blocking techniques, these linear operations can be represented as matrix addition and multiplication operations with various sizes. For instance, in convolutional neural network (CNN) models, the convolution and fully connected layers are the main linear operations that can be transformed into matrix multiplication by reshaping the input features and filters into two matrices. The dimensions of these matrices are determined by the width, height, and number of channels in the original convolution computation.

Given that each GEMM accelerator has its own computational and memory capabilities, matrices of different sizes--reshaped from linear operations in DNNs--are processed block-wise on the GEMM accelerator. In other words, the input and weight matrices are partitioned into smaller blocks to compute the output matrix, taking advantage of the GEMM accelerator's three-level memory hierarchy to minimize energy consumption and buffer access times [4]. The optimal block division is achieved by exploring data flows using a top-down approach: first addressing the stationary scheme, followed by spatial/temporal accesses, and finally determining the tile size to find the optimized data flow of linear operations. The term "stationary" refers to storing matrix data in global and local buffers for extended periods to maximize its reuse. Data reuse can be classified into temporal and spatial reuse. Temporal reuse involves reading data from off-chip DRAM in chronological order, sending it to multiple local buffers, and performing multiplication or addition operations on the partial sums in processing elements (PEs). Conversely, spatial reuse entails moving and processing data in parallel. Lastly, the tile size defines the data size for each movement and computation.

The above division uses a method similar to grid search to find this optimal block division. For example, given a matrix multiplication with dimension \((M\times K)\times(K\times N)\), we change the block size in the three dimensions (stationary, spatial/temporal accesses, and tile sizes) from 2 to 128 with stride 2, and use an early stage model to calculate the latency and energy consumption of GEMM accelerator. Then we will choose the optimal block size in three dimensions with the minimum latency under energy consumption constraints.

### Moving Nonlinear Operations to Matrix Multiplication

Addressing the nonlinear operations inherent in DNNs poses a significant challenge, as they cannot be easily mapped to standard matrix multiplications performed using GEMM accelerators, as described in the previous subsection. To overcome this issue, ASICs are specifically designed. These designs typically involve creating a predetermined number of functional units of various types, meticulously tailored to fulfill the nonlinear computational requirements of a specific deep neural network (DNN) model. While this approach yields highly efficient accelerators adept at handling particular networks, it falls short in terms of flexibility. In contrast, our proposed method emphasizes enhanced adaptability, allowing a wide range of DNNs to be efficiently executed on a single GEMM accelerator.

Offline Piecewise Linearization.In NeuralMatrix, continuous nonlinear operations are approximated using piecewise functions. This method involves dividing a nonlinear function of interest into smaller regions within a chosen interval and approximating each region with a simpler function, such
as a line or polynomial. Continuous piecewise linear (CPWL) approximations specifically utilize lines for approximation, ensuring that the endpoint of one region is the same as the starting point of the next region.

There are two primary advantages of employing CPWL approximation in NeuralMatrix. Firstly, classic GEMM accelerators can support CPWL without any hardware modifications, unlike some other approximation methods, such as look-up table (LUT) based nonlinear approximation, which require extra hardware resources. Secondly, alternative approximation methods like Taylor expansion or Chebyshev approximation necessitate considerable additional computations, which do not align with our ultimate goal of achieving ASIC-level computational efficiency.

Piecewise Linear Calculation.Technically speaking, piecewise linear operations from the above approximation are still nonlinear, and can not be directly mapped to GEMM. Therefore, we develop a three-step approach here that can handle piecewise linear calculations. Here we use piecewise linearized GELU as an illustrative example (in Fig. 2), but the same process can also be used to handle operations that are originally piecewise linear, such as ReLU and Leaky ReLU. The input data is \(X\) and the goal is to calculate \(Y=\text{GELU}_{approx.}(X)\), \(X,Y\in\mathbb{R}^{M\times N}\). The pre-calculated parameter \(k\) and \(b\) of each segment are pre-stored in off-chip DRAM, and indexed by the segment numbers.

Piecewise linear calculation follows the following steps: 1 We use a linear operator to calculate the segment matrix \(S\) for the input matrix \(X\). Each element in \(S\), e.g., \(S_{i,j}\) represents which segment its corresponding input value \(X_{i,j}\) falls into. The calculation of the segment matrix \(S\) is handled by the GEMM accelerator, and the output is further passed to the buffer/DRAM1; 2 The parameter \(k\) and \(b\) are aggregated and sent to the GEMM accelerator in the forms of slope matrix \(K\) and intercept matrix \(B\); 3 Finally, the GEMM accelerator performs element-wise calculations \(Y=X\cdot K+B\) to get the output matrix \(Y\). The other continuous nonlinear functions, such as softmax and layer normalization, can be computed by approximating inverse proportional, root, and exponential functions.

Footnote 1: When this piecewise linear approximation is calculated, the corresponding parameters are already prefetched from DRAM to the global buffers.

Parameter Overhead.NeuralMatrix introduces additional overhead to the original model, resulting from the extra parameters used to represent piecewise linear functions (e.g., \(k\), \(b\) tables). In this section, we provide a quantitative analysis of this aspect. Specifically, we focus on the scenario where the granularity is set to 0.25, as smaller granularity values lead to increased overhead in NeuralMatrix. Since we employ fixed granularity for the entire network, utilizing larger granularity values will proportionally decrease the parameter overhead. For instance, if the granularity is doubled to 0.5, the overhead will be reduced by half. Even with the smallest granularity in our experiments, the parameter overhead remains minimal (less than 0.7%). Therefore, in the following sections, we will utilize 0.25 as the default segment granularity.

### Approximation with Fine-Tuning

NeuralMatrix primarily focuses on improving the inference-time efficiency of fine-tuned DNNs on resource-limited platforms. One approach, which we referred to as the _post-finetuning_ method, involves standard fine-tuning of the original DNN. Following this, the DNN is transferred to GEMM with necessary approximated piecewise linearization, as detailed in the previous subsection.

In addition, NeuralMatrix can be seamlessly integrated with training, offering an alternative _pre-finetuning_ approximation approach. This technique involves mapping a pre-trained DNN to its approximated form before finetuning it on specific downstream tasks. The loss function used during finetuning remains unchanged from conventional finetuning, and standard automatic differentiation techniques can be employed for back-propagation. Both the post- and pre-finetuning methods yield final approximated DNNs with identical architectures, and consequently, have the same inference time cost on GEMM.

\begin{table}
\begin{tabular}{l r r r} \hline \hline DNN Model & ResNet-50 & BERT-base & GCN \\ \hline Extra Parameter Size (FP16) & 114KB & 49.2KB & 0.24KB \\ Extra Parameter Size (INT8) & 57KB & 24.6KB & 0.12KB \\ Normalized Parameter Size & 0.46\% & 0.01\% & 0.10-0.74\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: Parameter overhead in NeuralMatrix (granularity=0.25).
Any approximation method that maps DNNs, including nonlinear functions, to the GEMM accelerator will inevitably lead to a loss in computational accuracy, which in turn results in end-to-end inference accuracy loss. In the piecewise approximation of nonlinear operations, the accuracy of the approximation is dependent on the granularity of linear segments. Finer granularity contributes to higher accuracy but increases the number of parameters stored in the DRAM. In Section 4.2, we will demonstrate how to find a suitable tradeoff and select the appropriate granularity of linear segments to achieve both low memory cost and high inference accuracy for various downstream tasks.

## 4 Evaluation

We first verify inference accuracy of moving differnet DNNs to GEMM by the proposed NeuralMatrix. Next we compare with existing computation platforms to showcase the efficiency of NeuralMatrix. Finally, we test the scalibility of NeuralMatrix across different network sizes, including depth and width, as well as variant network models.

### Implementation Details

In this study, we utilized the state-of-the-art GEMM accelerator modeling framework (with 90-95% modeling accuracy) presented in [4], to estimate the GEMM accelerator performance and energy cost2 with 45nm chip fabrication technology node [22]. The architectural-level parameters, including the number of processing elements (PEs) and memory bandwidths of the GEMM accelerator, are determined by the rule of highest throughput per power on general matrix multiplication, given the constraints of the thermal design power cannot exceed 15W for mobile applications. The parameters are summarized in Table 2. We choose the output stationary data movement strategy as it avoids frequent data movements to and from memories and benefits the lowest energy consumption for large matrix multiplication [23].

Footnote 2: The energy cost is calculated through hardware module utilization multiplied by synthesized energy consumption sampled on different inputs.

To assess the computation latency and power consumption of CPUs and GPUs, we conducted tests on the Intel i7-11700 CPU, NVIDIA 3090Ti GPU, and Jetson Orin SoC, utilizing an IT9121 power analyzer. For the existing FPGA and ASIC solutions, we gathered relevant data from published papers. Our analysis focuses on network inference latency (including throughput and relative speedup) as well as power consumption (energy usage). It is worth noting that our evaluation metric does not employ floating-point operations per second (FLOPS), as it solely indicates the computational capability of the processor. However, there exists a significant disparity between the computation supported by the processor and the actual utilization by DNNs.

NeuralMatrix moves the versatile DNN models to GEMM, and the entire network was computed by general matrix multiplications and additions. We experiment with three main DNN categories: BERT, ResNet and GCN. We evaluate the performance of our models using standard benchmarks for each network type, considering both floating-point (FP16) and integer (INT8) quantization of each network model. For CNN-based ResNet-50, we use CIFAR-10 [24], CIFAR-100 [24], Mnist-Fashion [25], and QMNIST [26] datasets. For transformer-based BERT, we use the General Language Understanding Evaluation (GLUE) benchmark [27], which consists of nine language understanding tasks, ranging in dataset sizes, text genres, and difficulty levels. For GNN-based GCN, we use Cora [28], Citeseer [29], Pubmed [30], and Reddit [31] datasets.

### Inference Accuracy

Before demonstrating the computational efficiency advantage of NeuralMatrix, we first empirically verify its inference accuracy through three popular DNN architecture categories and seventeen tasks of different natures. We ensure that applying NeuralMatrix with the appropriate setup only compromises ignorable accurate loss compared with the original models' final performance. Fig. 3 displays the final inference accuracy of various DNN architectures on multiple benchmark datasets. In this experiment, we select some of the most well-known pre-trained DNNs in each category,

\begin{table}
\begin{tabular}{c|c c c c} \hline \hline Parameters & **Total PE Numbers** & **Cluster Numbers** & **Stationary Scheme** & **Frequency** \\ Value & 1024 & 8 & Output stationary & 1 GHz \\ \hline Parameters & **L1 buffer Size** & **L2 buffer Size** & **NoC Bandwidth** & **DRAM Bandwidth** \\ Value & 4 KB & 1 MB & 128 Gbps & 32 Gbps \\ \hline \hline \end{tabular}
\end{table}
Table 2: The parameters of our GEMM accelerator.
such as ResNet-50 [32], BERT-base [33], and GCN [34], to represent CNN, transformer, and GNN, respectively. Although we experimented with 17 benchmark datasets, due to space limitations, we only showcase four benchmarks for each DNN category and include the rest in the supplementary materials. The dashed lines in each figure represent the inference accuracy of the original floating-point and quantization DNN models, which can be considered as the performance upper-bound given the pre-trained network. The vertical bars in the sub-figures illustrate different NeuralMatrix variants. Specifically, the leftmost bars represent the post-finetuning performances, while the middle and right bars indicate the pre-finetuning performance using floating-point and INT8 quantization, respectively.

From the figures, we observe that the best-performing NeuralMatrix attains comparable performance to the original DNNs in every benchmark across different DNN categories. We find that pre-finetuning consistently achieves better performance than the post-finetuning approach, possibly because the network during finetuning is identical to the final inference ones. Thus, we recommend using the pre-finetuning method when transitioning DNNs to GEMMs. As the granularity increases from 0.25 to 1.0, we notice inference accuracy declines across all NeuralMatrix variants, which is expected since larger granularity incurs higher approximation errors. Generally, accuracy only begins to drop gradually when the granularity of exceeds 0.5 with floating-point and 0.25 with INT8 quantization. This demonstrates that a reasonably small granularity can achieve desirable performance.

Regarding specific DNN models, we observe that for both BERT and ResNet, the performance gap between different NeuralMatrix variants increases as the baseline performance decreases. This suggests that one can choose a larger granularity for easier tasks but a smaller one for more difficult tasks. In contrast, the GCN models do not exhibit significant differences among the baseline and various NeuralMatrix variants, possibly because GCNs are typically shallower. We provide further analysis regarding DNN depth and performance in Section 4.4 below.

### Benefits of Computation and Energy Efficiency

This section presents the potential computation and energy cost savings that can be reaped by moving DNNs to matrix multiplication and executing it on a GEMM accelerator. We compare the inference latency and cost of running neural networks on various types of processing units, including general-purpose CPUs, GPUs, SoCs, application-specific FPGA and ASIC designs, as well as the

Figure 3: End-to-end inference accuracy of different tasks using different DNN architectures.
NeuralMatrix on a GEMM accelerator. For the GNN, we only list the GNN runs on general-purpose processors as we are not able to find the standard ASIC designs for GNN who has numerous variants.

Table 3 provides a comparison of the inference latency and energy cost of moving ResNet-50 and BERT to general matrix multiplication using various types of processors. Moving ResNet-50 to general matrix multiplication achieves 9.64\(\times\) and 1.42\(\times\) improvements on latency with 11.7\(\times\) and 13.71\(\times\) reduced power consumption compared with the general-purpose CPU and GPU. While moving BERT to general matrix multiplication can achieve 8.53\(\times\) and 1.48\(\times\) improvements on latency with 9.48\(\times\) and 11.1\(\times\) reduced power consumption compared with the general-purpose CPU and GPU. If the throughput per power is regarded as the metrics for computation efficiency, NeuralMatrix achieves up to 113 \(\times\) and 19.44 \(\times\) improvements compared with CPU and GPU. Furthermore, compared with the FPGA-based ResNet-specific and BERT-specific designs, moving ResNet-50 and BERT to matrix multiplication also improves the computation efficiency by 2.32 - 3.50 times. Also, we must admit that part of the benefits, compared to FPGA-based BERT-specific designs, comes from the inherent superiority of the ASIC-based operation over the LUT-based operations in clock frequency and power consumption. Finally, compared with the ResNet-specific and BERT-specific ASIC designs, the same levels (i.e., 86.0% - 310.6%) of computation efficiency are achieved given the generality of running versatile on the same GEMM accelerator.

### Scalability Across Network Sizes

Further, we experimentally test and compare the inference accuracy of NeuralMatrix across different network sizes, including depth and width, as well as variant network models. The inference accuracy of NeuralMatrix is summarized in Table 4 for different sizes of ResNet, BERT, GCN, respectively. From the accuracy results, it is evident that the accuracy of NeuralMatrix is relatively robust to various network sizes and variant models of classic networks.

Figure 4 illustrates the computation efficiency (throughput per power) of NeuralMatrix for different network sizes and variants on the GEMM. The plots also indicate the related ASIC designs for these

\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{Type} & \multirow{2}{*}{Processor} & \multirow{2}{*}{Models} & \multirow{2}{*}{Size} & \multicolumn{3}{c}{FP16} & \multicolumn{3}{c}{INTS} \\ \cline{5-12}  & & & Original & NeuralMatrix & & Original & NeuralMatrix & & Original & NeuralMatrix \\ \hline \multirow{8}{*}{CNN/CIFAR-10} & ResNet-18 & 44.8 MB & 95.02 & 92.87 (-2.13) & 94.66 & 92.12 (-2.54) \\  & ResNet-34 & 85.3 MB & 96.12 & 94.85 (-1.27) & 96.10 & 94.43 (-1.67) \\ \cline{1-1}  & ResNet-50 & 90.1 MB & 96.18 & 95.67 (-1.7) & 96.13 & 94.43 (-2.05) \\ \cline{1-1}  & ResNet-101 & 163 MB & 97.13 & 95.67 (-1.51) & 96.44 & 94.82 (-1.62) \\ \cline{1-1}  & ResNet-152 & 223 MB & 97.29 & 95.12 (-2.17) & 96.87 & 94.38 (-2.49) \\ \hline \multirow{8}{*}{Transformers/SST-2} & ALBERT & 47.4 MB & 92.50 & 88.42 (-4.13) & 89.76 & 85.28 (-4.48) \\  & DistilBERT & 263 MB & 90.41 & 88.65 (-1.35) & 88.19 & 86.24 (-1.95) \\ \cline{1-1}  & BERT-Base & 436 MB & 92.32 & 92.32 (-0.00) & 92.24 & 92.07 (-0.25) \\ \cline{1-1}  & BERT-Large & 1340 MB & 93.46 & 93.02 (-0.34) & 93.12 & 92.20 (-0.92) \\ \hline \multirow{8}{*}{GNN/CORA} & GCN (L=1) & 94.0KB & 72.90 & 72.34 (-0.56) & 72.37 & 71.84 (-0.53) \\ \cline{1-1}  & GCN (L=2) & 95.6KB & 84.28 & 84.31 (+0.03) & 83.95 & 84.02 (+0.07) \\ \cline{1-1}  & GCN (L=3) & 99.4KB & 83.44 & 84.38 (+0.04) & 84.21 & 84.11 (-0.10) \\ \cline{1-1}  & GCN (L=6) & 113.9KB & 81.18 & 81.11 (+0.07) & 80.86 & 80.92 (+0.06) \\ \cline{1-1}  & GCN (L=9) & 128.3KB & 80.53 & 80.58 (+0.05) & 79.82 & 79.91 (+0.09) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Inference accuracy (%) of NeuralMatrix compared with original DNN models of different categories and sizes. We use CIFAR-10, SST-2 and CORA datasets for CNN, Transformer and GNN respectively.
networks (ResNet and BERT) [35; 36; 37; 38; 39; 40; 41; 42]. Compared to ASIC designs for networks of different sizes, NeuralMatrix can achieve the same level of computation efficiency. For the network with smaller sizes, the computation efficiency of NeuralMatrix increases as the networks become wider and deeper. Network width has a greater impact on computation efficiency compared to depth. Later, the computation efficiency reaches the peak and stable range for the network with large sizes. According to this design space exploration, the NeuralMatrix will be more effective for the large neural network, especially the wider networks. This is because a wider neural network will be generated to a larger matrix and allows more parallel computation to be performed simultaneously by the parallel processing elements in the GEMM accelerator. We can conclude that by transforming the computation of the entire neural network to general matrix multiplication, NeuralMatrix enables versatile neural networks to be efficiently executed by one GEMM accelerator. Meanwhile, NeuralMatrix demonstrates its superior efficiency for large neural networks, which can fully utilize its PEs after being transformed to matrix multiplication by NeuralMatrix.

## 5 Discussion and Limitations

In this study, we concentrated on three widely-used DNN backbone models and employed fixed approximation granularities for the entire DNN as a demonstrative example to highlight the benefits of transitioning entire neural networks to general matrix multiplications. Nevertheless, we believe that NeuralMatrix can be applied to broader categories of network architectures as well. Moreover, by examining different nonlinear functions and their positions, it may be feasible to further minimize the accuracy loss of NeuralMatrix by employing varying approximation granularities [43]. Due to time and space constraints, we only implemented and tested the piecewise linear approximation method; however, alternative approximation methods might potentially enhance network inference accuracy at the expense of increased computation cost. Furthermore, our evaluation of scalability across network sizes revealed that larger (i.e., wider) networks demonstrate greater computation efficiency gains. Moving forward, we also envision the proposed NeuralMatrix being effectively utilized not only in the inference process but also in the training process of large models in the future.

## 6 Conclusion

Current hardware accelerators based on application-specific integrated circuits (ASIC) are often limited to efficiently supporting specific deep neural network (DNN) applications. To overcome this limitation and enable versatile DNNs on a single hardware accelerator, we introduce NeuralMatrix. This approach transitions entire DNNs to matrix multiplication, which can then be executed by general matrix multiplication accelerators. NeuralMatrix utilizes three mainstream DNN backbone models - CNN, transformer, and GNN - along with their variant models as illustrative examples. Our pre-finetuning training reveals that the shift to matrix multiplication incurs negligible inference accuracy loss. The evaluation demonstrates that NeuralMatrix can attain ASIC-level computation and energy efficiency on general-purpose matrix multiplication accelerators. Consequently, this enables the efficient support of a broad range of DNNs on a single hardware accelerator.

Figure 4: The computation efficiency (throughput per power) of NeuralMatrix with different network categories under different widths and depths.