FPGA Resource-aware Structured Pruning for Real-Time Neural Networks
Neural networks achieve state-of-the-art performance in image classification,
speech recognition, scientific analysis and many more application areas. Due to
the high computational complexity and memory footprint of neural networks,
various compression techniques, such as pruning and quantization, have been
proposed in literature. Pruning sparsifies a neural network, reducing the
number of multiplications and memory. However, pruning often fails to capture
properties of the underlying hardware, causing unstructured sparsity and
load-balance inefficiency, thus bottlenecking resource improvements. We propose
a hardware-centric formulation of pruning, by formulating it as a knapsack
problem with resource-aware tensor structures. Evaluated on a range of tasks,
including sub-microsecond particle classification at CERN's Large Hadron
Collider and fast image classification, the proposed method achieves reductions
ranging between 55% and 92% in the DSP utilization and up to 81% in BRAM
utilization.
Benjamin Ramhorst, Vladimir Loncar, George A. Constantinides
# FPGA Resource-aware Structured Pruning

###### Abstract

Neural networks achieve state-of-the-art performance in image classification, speech recognition, scientific analysis and many more application areas. With the ever-increasing need for faster computation and lower power consumption, driven by real-time systems and Internet-of-Things (IoT) devices, FPGAs have emerged as suitable devices for deep learning inference. Due to the high computational complexity and memory footprint of neural networks, various compression techniques, such as pruning, quantization and knowledge distillation, have been proposed in literature. Pruning sparsifies a neural network, reducing the number of multiplications and memory. However, pruning often fails to capture properties of the underlying hardware, causing unstructured sparsity and load-balance inefficiency, thus bottlenecking resource improvements. We propose a hardware-centric formulation of pruning, by formulating it as a knapsack problem with resource-aware tensor structures. The primary emphasis is on real-time inference, with latencies in the order of 1\(\upmu\)s, accelerated with hls4ml, an open-source framework for deep learning inference on FPGAs. Evaluated on a range of tasks, including real-time particle classification at CERN's Large Hadron Collider and fast image classification, the proposed method achieves a reduction ranging between 55% and 92% in the utilization of digital signal processing blocks (DSP) and up to 81% in block memory (BRAM) utilization.

 FPGA, Deep Learning, Pruning

## I Introduction

Modern neural networks are associated with high computational complexity and memory footprint, linked to the high dimensionality of weight tensors. With the ever-increasing need for faster computation and lower power consumption, driven by real-time systems and Internet-of-Things (IoT), neural networks must be adapted to reduce their complexity and memory footprint. A common method for network compression is pruning, the process of sparsifying a neural network, by setting some of its weights to zero. LeCun _et al._[1] first proposed the idea of Optimal Brain Damage: for a typical neural network, it is possible to remove over half of the weights, with no loss in accuracy. Since then, pruning has been extensively studied in the literature [2, 3, 4] and evaluated on a range of problems and hardware platforms. However, to achieve maximum improvements in resource utilization and computational complexity, pruning must consider the underlying hardware.

hls4ml [5, 6] is an open-source library for real-time inference of neural networks on FPGAs, built on top of high-level synthesis (HLS). Primarily intended for real-time deep learning at CERN, hls4ml has become a powerful tool evaluated on a range of tasks: particle physics [6, 7, 8], autonomous vehicles [9] and fast image classification [10]. To meet the strict latency constraints of experiments at CERN, hls4ml employs a full on-chip design with high parallelism. However, as with any high-performance system, neural networks accelerated with hls4ml are susceptible to high resource utilization. Previous compression efforts with hls4ml included quantization-aware training with QKeras [11], up to extreme, binary and ternary precisions [12], with limited emphasis on pruning.

We propose an FPGA resource-aware structured pruning algorithm, able to capture, at training time, the underlying mapping to digital signal processing blocks (DSPs) and block random access memory (BRAM). Groups of weights are iteratively removed through constrained optimization, formulated as a knapsack problem. Evaluated on particle physics benchmarks and fast image classification tasks, the proposed method achieves a reduction ranging between 55% and 92% in DSP and up to 81% in BRAM utilization while maintaining inference latencies in the order of a few microseconds. The methodology described in this research has been integrated with hls4ml and open-sourced with an Apache 2.0 license1, allowing long-term contributions. Key contributions of our research include:

Footnote 1: GitHub page: [https://github.com/fastmachinelearning/hls4ml/tree/hardware-pruning](https://github.com/fastmachinelearning/hls4ml/tree/hardware-pruning)

* A structured pruning algorithm, derived from the mapping of model weights to DSP and BRAM blocks.
* Formulation of pruning as a knapsack problem, aimed at minimizing network loss and resources.
* An open-source tool for resource-aware pruning of Keras and TensorFlow models, integrated with hls4ml, enabling real-time inference.
## II Background

### _Pruning_

Modern neural networks have billions of weights, leading to long inference latencies and high resource utilization. Pruning is a network compression technique aimed at producing a sparse neural network. First proposed by LeCun _et al._[1], it was applied to modern neural networks by Han _et al._[13], implementing an iterative, magnitude-based pruning algorithm. The networks are trained with regularization, shifting weights towards zero. Weights below a certain magnitude are pruned and the remaining weights are updated, retraining the network. Han _et al._ implemented unstructured pruning, removing individual weights from the network. Through unstructured pruning, high sparsities are achievable, but, with little improvements in resource utilization and inference latency [2, 4]. Sparse, unstructured matrices, require additional encodings, such as compressed sparse row (CSR) or compressed sparse column (CSC), incurring additional overheads in memory utilization. In HLS, unstructured multiplications can only be optimized by the compiler if loops are fully unrolled; otherwise, multiplications by zero are executed during inference. On the other hand, structured pruning [14, 15, 16], achieves improvements in resource utilization and inference latency by removing entire structures within weight tensors. Compared to unstructured pruning, for the same level of sparsity, structured pruning achieves a lower accuracy.

### _hls4ml_

High-energy physics experiments, such as those undertaken at CERN, collide sub-atomic particles near the speed of light. Current experiments aim to fully characterize the Higgs boson as well as search for physics beyond the Standard Model, including dark matter [6]. By accelerating sub-atomic particles near the speed of light, hundreds of terabytes of data are generated each second. Triggering is the process of analyzing and selecting only the most important data. Due to the extreme frequencies of particle collisions, a strict latency constraint is imposed, ranging from 100ns to a few microseconds, depending on the experiment. Therefore, specialized hardware, including ASICs and FPGAs, is used for data processing. An early attempt of deep learning in triggering was achieved with hls4ml [6] and has been shown effective in a range of studies, including the acceleration of convolutional autoencoders for anomaly detection [7], recurrent neural networks [8] and graph neural networks [17].

hls4ml allows fine-grained tuning of the hardware configuration, through the following properties:

* _Reuse factor_ (RF): a layer-wise variable controlling the trade-off between inference latency and resource utilization, indicating the number of multiplications executed in parallel, as shown in Fig. 1.
* Precision: all variables are represented using a fixed-point format, with a tunable precision. By using low-precision fixed-point arithmetic, instead of floating-point, hls4ml aims to reduce inference latency and resource utilization.
* I/O type: streaming or parallel. Parallel I/O is suitable for fully connected (FC) networks targeting ultra-low latencies, as activation tensors are processed in parallel and stored in registers. Streaming I/O is suitable for large-scale convolutional neural networks (CNNs), as activation tensors are processed sequentially and stored in BRAM. Streaming I/O reduces resource consumption at the expense of latency.
* _Latency_ or _Resource_. In the _Latency_ strategy, weights are stored in registers, enabling a faster inference at the expense of resource utilization, making it unsuitable for larger models. In the _Resource_ strategy, weights are stored in BRAM, reducing resource consumption and increasing inference latency.

In the _Latency_ strategy, unstructured pruning is optimal, as multiplications by zero are optimized by Vivado HLS (unrolled loops and weights are stored in registers). While suitable for small networks, the _Latency_ strategy does not scale well, due to Vivado partition and unroll limits. Our work focuses on pruning in the _Resource_ strategy, as it enables the acceleration of a wide range of networks. In _Resource_ strategy, weights are transposed before hardware synthesis, ensuring all the weights processed in the same clock cycle can simultaneously be read from BRAM. An example mapping from weight tensors to BRAM is given in Fig. 2. Algorithm 1 summarizes the implementation of matrix-vector multiplication in _Resource_ strategy, which forms the baseline for FC layers and is used for kernel multiplication in convolutional (CONV) layers. In Section III-A, the algorithm is used to determine the mapping between weight tensors and DSP blocks, a key component of resource-aware pruning.

Fig. 1: Variations of RF and the impact on resource utilization [6].

Fig. 2: Example mapping of weights to BRAM.
### _Related work_

Several works proposed hardware-aware pruning algorithms suitable for FPGA inference. Yu _et al._ proposed Scalpel [18], a pruning algorithm customized to the level of parallelism. On micro-controllers with a single instruction, multiple data (SIMD) processor, the authors split the weights into aligned groups, each group of size equal to the SIMD input size. Then, groups below a certain threshold were removed and the network was retrained. Kang [19] and Li & Louri [20] proposed pruning methods so that each processing element (PE) processes the same number of zeros, solving the load-balance problem. Ren _et al._[21] proposed a hardware-aware pruning algorithm, based on the alternating direction method of multipliers (ADMM). Through ADMM, the authors formulate pruning as a discrete optimization problem, restricting the weight tensors to an admissible set, depending on the target hardware. Zhannong _et al._[22] propose pattern pruning for CONV kernels, by noticing that the number of combinations in 3x3 kernels is 512. Through unstructured pruning, the algorithm selects the dominant pattern and projects the remaining onto the selected set, thus encoding each filter as a 7-bit value. Similar work was undertaken by PACA [23] and PatDNN [24], but, through ADMM optimization. However, pattern pruning is only applicable to large-scale neural networks, with many, 3x3 filters. Furthermore, for 5x5 filters, the proposed method does not scale.

A complementary method to pruning is network quantization. FINN [25, 26], is an open-source framework for generating custom dataflow-style architectures for quantized networks. Faraone _et al._[27] proposed a resource-aware, filter-level pruning algorithm for quantized neural networks accelerated with FINN. Previous compression efforts with hls4m1 primarily focused on quantization-aware training with QKeras [11], up to extreme, binary and ternary precisions [12]. Wang _et al._[28] proposed LUTNet, a hardware-software framework for acceleration of sparse, binary neural networks using K-LUT. In LUTNet, networks are trained and pruned with floating-point precision, before binarization and acceleration using K-LUTs. In a follow-up work, the authors, proposed Logic Shrinkage [29]: a fine-grained pruning methodology for selecting the parameter K and reducing LUT utilization, evaluated on large-scale CNNs.

Our method differs from the above in several ways. While similar to Kang [19] and Li & Louri [20], our method considers mapping to DSPs as well as BRAM. Shen _et al._[30] proposed a latency-aware structured pruning method, by formulating pruning as a knapsack problem. While similar to our work, we focus on structured resource-aware pruning targeting FPGA inference, compared to the work of Shen _et al._, focusing on GPU inference latency. Furthermore, many of the proposed methods focused on pruning large-scale CNNs, such as VGG or ResNet. Our method focused on pruning smaller, shallower methods suitable for ultra-low latencies. Finally, our implementation is open-sourced and integrated with hls4m1, enabling an end-to-end flow of resource-aware pruning, quantization-aware training, through the interface with QKeras [11], and real-time inference.

## III Resource-aware pruning

### _Resource-aware tensor structures_

To maximize resource savings of accelerated neural networks, we consider the reduction of DSP and BRAM utilization, by pruning all the weights processed by the same multiplier or block of memory. The mapping between weights and DSP blocks is deterministic from the RF. Given a weight matrix, weights processed by the same DSP block can be obtained by transposing and flattening the matrix into a vector. The vector is split into sub-vectors of length RF, with each sub-vector representing the weights processed by the same DSP. Illustrated in Fig. 3, pruning weights \((w_{1},w_{5},w_{9})\), or \((w_{2},w_{6},w_{10})\), or \((w_{3},w_{7},w_{11})\), or \((w_{4},w_{8},w_{12})\) would achieve a reduction of one DSP block, further referred to as DSP-aware pruning.

DSP-aware pruning can readily be extended to optimizing memory utilization. Weights processed by the same DSP are stored as subsequent words in the same block. In hls4m1, BRAM is typically implemented as 1K x 36. Weights are quantized before deployment to a lower precision, so each BRAM will store weights processed by several neighboring DSPs. As an example, weights quantized to 18 bits would map to two DSP blocks for each block of RAM. To optimize BRAM utilization, given weight precision (\(P\)), \(C\) consecutive
groups of weights processed by the same DSP block need to be pruned, given by Equation 12.

Footnote 2: \(\equiv\) is the congruence modulo operator.

\[C=\begin{cases}\frac{36}{P}&\text{if }36\equiv 0\bmod P\\ \lceil\frac{2.36}{P}\rceil&\text{otherwise}\end{cases} \tag{1}\]

If the precision is a factor of BRAM width, each pruning step ensures one block of RAM is saved. Otherwise, pruning must capture weight with width equal twice the width of BRAM, ensuring at least one block of RAM is removed. Therefore, to aid pruning and mitigate an increased network loss, the precision should be set to a factor of 36. In subsequent sections, we refer to BRAM-aware pruning as multi-dimensional pruning since each pruning step removes \(C\) DSP blocks and one block of RAM.

### _Pruning as a knapsack problem_

hls4ml supports heterogeneous configuration of layers, allowing fine-grained tuning of the hardware-configuration: RF, precision and strategy. Furthermore, Vivado aims to implement multiplications for precisions lower than 10 bits through LUTs, rather than DSP blocks. As a result of these factors, pruning becomes a non-trivial optimization problem, aimed at minimizing network loss and overall resource consumption. We solve this problem by transforming it to a knapsack problem, similar to the work by Shen _et al._[30] on latency-aware pruning for GPUs.

Given a neural network, \(\mathcal{N}\), let \(\mathbb{W}=\{\mathbf{w}_{1},\ldots,\mathbf{w}_{n}\}\) be the set of resource-aware vectors, with each vector containing network weights mapped to the same resource (BRAM or DSP). Without pruning, the set \(\mathbb{W}\), contains all the network weights, grouped per resource mapping. We define \(R:\mathbb{R}^{k}\rightarrow\mathbb{R}^{m}\) as the resource estimation function for resource-aware structures. The resource estimation function has no explicit format, but can be calculated by considering RF, precision and strategy. As an example, consider the following cases:

* Layers quantized to 18 bits, with BRAM-aware structures would map to two DSP blocks and one block of RAM, for every resource-aware structure, \(\mathbf{w}_{i}\).
* Layers quantized to 9 bits, with BRAM-aware structures would map to zero DSP blocks3 and one block of RAM (with \(C=4\) consecutive groups), for every resource-aware structure, \(\mathbf{w}_{i}\). Footnote 3: Multiplication is implemented using LUT for precisions smaller than 10 bits.
* Layer quantized to 16 bits, with DSP-aware structures would map to one DSP, for every resource-aware structure, \(\mathbf{w}_{i}\).

The resource estimation function is, in general, a vector-valued function, considering \(k\) inputs (dimensionality of resource-aware structures) with \(m\) outputs. The dimensionality of resource-aware structures varies across layers, determined from the RF, precision and strategy. Considering \(m\) outputs enables modeling of several hardware resources. Given training data, \(\mathcal{D}\), and the network loss function, \(\mathcal{L}(\mathbb{W};\mathcal{D})\), we formulate pruning as a multi-objective optimization problem of selecting a subset of weights, \(\hat{\mathbb{W}}\subseteq\mathbb{W}\), minimizing network loss and overall resource utilization:

\[\min_{\mathbb{W}\subseteq\mathbb{W}}\ \mathcal{L}(\hat{\mathbb{W}},\mathcal{D}), \sum_{\mathbf{w}_{i}\in\hat{\mathbb{W}}}R(\mathbf{w}_{i}) \tag{2}\]

We transform Equation 2 by minimizing the loss, while restricting the overall resource utilization, using the \(\epsilon-\)constraint method [31]:

\[\min_{\mathbb{W}}\ \mathcal{L}(\hat{\mathbb{W}},\mathcal{D})\] (3) subject to \[\sum_{\mathbf{w}_{i}\in\hat{\mathbb{W}}}R(\mathbf{w}_{i})\preceq \mathbf{c}\] \[\hat{\mathbb{W}}\subseteq\mathbb{W}\]

The problem of minimizing network loss obtained from selecting a subset of network weights is often approximated by selecting the subset of weights with the highest magnitude [13, 14, 15, 16], followed by fine-tuning of the selected weights:

\[\max_{\hat{\mathbb{W}}}\ \sum_{\mathbf{w}_{i}\in\hat{\mathbb{W}}}\frac{ \left\|\mathbf{w}_{i}\right\|}{\max_{\mathbf{w}_{i},\mathbf{w}_{j}\in L} \left\|\mathbf{w}_{j}\right\|}\] (4) subject to \[\sum_{\mathbf{w}_{i}\in\hat{\mathbb{W}}}R(\mathbf{w}_{i})\preceq \mathbf{c}\] \[\hat{\mathbb{W}}\subseteq\mathbb{W}\]

Fig. 3: Mapping of weights to DSP with RF=3.
Considering the norm of each structure would lead to bias towards highly-dimensional structures or layers with large weights. Therefore, the norm of every structure, \(\mathbf{w}_{i}\), is normalized, with respect to the largest norm in the same layer, \(L\). To solve the combinatorial optimization problem in Equation 4, we consider the 0-1 knapsack problem:

\[\max_{\mathbf{x}}\mathbf{v}^{T}\mathbf{x}\] (5) subject to \[\mathbf{u}^{T}\mathbf{x}\leq c\] \[\mathbf{x}\in\{0,1\}^{n}\]

The problem considers \(n\) items, each with value, \(v_{i}\) and weight \(u_{i}\). The objective of the knapsack problem is to select a subset of the items, maximizing overall value while keeping the selected weight under a certain limit. For resource-aware pruning, each item is the resource-aware tensor structure, \(\mathbf{w}_{i}\). Item values are equal to the normalized magnitude, \(\frac{\|\mathbf{w}_{i}\|}{\max_{\mathbf{w}_{i}}\mathbf{w}_{j}\in L}\frac{\| \mathbf{w}_{j}\|}{\|\mathbf{w}_{j}\|}\). Item weights are equal to the resource consumption, \(R(\mathbf{w}_{i})\). By solving the knapsack problem, the most important weights, maximizing accuracy, are selected, while the rest are discarded, in order to meet the resource budget. The solution of the knapsack problem is such that:

\[x_{i}=\begin{cases}0&\text{if }\mathbf{w}_{i}\text{ pruned}\\ 1&\text{otherwise}\end{cases} \tag{6}\]

The knapsack problem readily extends to multiple dimensions (MDKP):

\[\max_{\mathbf{x}}\mathbf{v}^{T}\mathbf{x}\] (7) subject to \[\mathbf{U}\mathbf{x}\preceq\mathbf{c} \tag{8}\] \[\mathbf{x}\in\{0,1\}^{n}\]

MDKP enables the modeling of multiple resource objectives. In our case, we model DSP and BRAM. The 1-dimensional knapsack problem can effectively be solved using a fully polynomial-time approximation scheme (FPTAS) with dynamic programming, as well as branching methods. The MDKP has no FPTAS, unless \(P=NP\)[32]. We find that, for problems of our size, it can be effectively solved using branch-and-cut methods available from the OR-Tools library [33].

### _End-to-end algorithm_

We implement magnitude-based iterative pruning, described in Algorithm 2. The algorithm requires a pre-trained neural network, \(\mathcal{N}\), training and validation data, the target sparsity \(\mathbf{s_{T}}\) and hardware and training configuration. The target sparsity is a vector, with dimensionality equal to the number of resources considered. In our case, we consider DSP-aware pruning (one-dimensional) and two-dimensional DSP- and BRAM-aware pruning. Pruning is implemented iteratively, until the target resource consumption is achieved, or, network performance on the validation set drops below the relative tolerance. Training and hardware configuration include layer-wise RF, precision, strategy, descent algorithm, batch size _etc._

Key steps of resource-aware pruning are:

* Regularization: a resource-aware regularization loss is added to the network loss. Through regularization, the objective is to shift weights sharing the same hardware resource towards zero. Similar to Wen _et al._[16], we implement group regularization. However, unlike Wen _et al._, weights are not grouped per filter; instead, they are grouped per hardware resource.
* Initialization: Resource-aware structures are identified. The initial sparsity4 is set to zero, and, the baseline resource utilization is obtained as the sum of the resource utilization of all the structures. Finally, the baseline performance on the validation set is calculated. Footnote 4: Sparsity is relative to the baseline resource utilization, not the total number of parameters.
* Iterative pruning: The network is pruned iteratively, by solving the MDKP, which selects the most important resource-aware structures, given a resource constraint. The constraint is iteratively reduced, by updating the sparsity, through a user-defined function, \(f(\cdot)\). In our case, sparsity is incremented by a constant step size. From the MDKP solution, a subset of the network weights, \(\hat{\mathbb{W}}\) are selected and fine-tuned, while the remaining weights, \(\mathbb{W}/\hat{\mathbb{W}}\) are set to zero. Fine-tuning is continuously done with regularization, pushing weights toward zero before updating sparsity. Footnote 4: Sparsity is relative to the baseline resource utilization, not the total number of parameters.

```
0: Neural network \(\mathcal{N}\), Training data \(\mathcal{D}\), Validation data \(\mathcal{D}\), Target sparsity \(\mathbf{s_{T}}\in\mathbb{R}_{+}^{m}\), Tolerance \(\epsilon\), Hardware and training configuration
```

Identify resource-aware structures \(\mathbb{W}=\{\mathbf{w}_{1},\ldots,\mathbf{w}_{n}\}\)\(\mathbf{s}\leftarrow\mathbf{0}\)\(\mathbf{R}_{B}\leftarrow\sum_{i=1}^{n}R(\mathbf{w}_{i})\)\(b\leftarrow\texttt{evaluate($\mathcal{N}$;$\mathbb{W}$)}\)\(p\gets b\)while\(\mathbf{s}\preceq\mathbf{s_{T}}\wedge p\geq\epsilon\cdot b\)do\(v_{i}\leftarrow\frac{\|\mathbf{w}_{i}\|}{\max_{\mathbf{w}_{i}}\mathbf{w}_{j} \in\|\mathbf{w}_{j}\|}\quad i=1,...,n\)\(R_{i}\gets R(\mathbf{w}_{i})\quad i=1,...,n\) Solve MDKP with \(v_{i}\), \(\mathbf{U}_{:,i}=R_{i}\) and \(\mathbf{C}=(1-\mathbf{s})\odot\mathbf{R}_{B}\)5 Solving MDKP returns a set of selected weights, \(\hat{\mathbb{W}}\) Fine-tune pruned network \(\mathcal{N}(\hat{\mathbb{W}})\) with regularization \(p\leftarrow\texttt{evaluate($\mathcal{N}$;$\hat{\mathbb{W}}$,$\hat{D}$)}\)\(\mathbf{s}\gets f(\mathbf{s})\) endwhile ```

**Algorithm 2** Resource-aware network pruning

To maximize resource savings from resource-aware pruning, we implement an automated code generation framework, to ignore DSP blocks processing constant zeros. This optimization is not done by the HLS compiler. The framework will generate HLS RTL tailored for every layer, with multiplications by
zero ignored, with, otherwise equivalent logic to the baseline implementation of matrix multiplication. Finally, hls4m1 uses task-level pipelining (i.e. pragma HLS dataflow) in parallel networks, inserting additional first-in, first-out (FIFO) channels, enabling task-level concurrency and reducing resource (LUT) utilization. However, pruning reduces the overall complexity of the network, and, therefore, we implement system-level pipelining (i.e. pragma HLS pipeline) instead. As confirmed by the results in Section IV, system-level pipelining reduces latency, and, the increase in logic utilization is mitigated through pruning. This optimization does not apply to streaming convolutions. The end-to-end flow for resource-aware pruning is shown in Figure 4.

## IV Results

### _Experimental setup_

To evaluate the effects of resource-aware pruning, three benchmark models are considered, with varying complexity, depth and application. Since hls4m1 is intended for ultra-low latency applications, we focus on application-specific, shallow networks with the following benchmarks, summarized in Table I:

* Jet classification for triggering. Jets are collimated showers of sub-atomic particles that occur due to the decay of particle collisions [6]. By considering 16 jet features (multiplicity, momentum, _etc._), the particles can be classified into one of five categories, the interesting observations: \(W\) boson, \(Z\) boson, \(t\) quark or background events, quark \(q\) and gluon \(g\). The FC architecture is derived from Duarte _et al._[6] using the data set from Zenodo [34].
* Street View House Numbers (SVHN) [35] classification, with the CNN architecture described by Aarrestad _et al._[10]. The architecture described in [10] was optimized for low-latency inference on the non-trivial digit dataset, by limiting the number of layers and weights.
* Fashion MNIST [36] classification, with a LeNet-like architecture [37]. The input size was changed from 32x32 to 28x28 to match the Fashion MNIST dataset. The 5x5 kernels were replaced with 3x3 kernels and ReLU activation [38] was added after every FC and CONV layer.

The models were trained using Keras [39] with a TensorFlow backend [40], by minimizing categorical cross-entropy using Adam optimization [41]. Models are pruned with constant increments in sparsity until the relative drop in validation accuracy exceeds 2%. The models are synthesized using Vivado and Vivado HLS 2020.1 with hls4m1 0.7.1 deblinium, reporting resource utilization post place and route. The target device is Xilinx Virtex UltraScale+ XCVU9P (xcvu9p-flgb2104-2-e). Latency is reported following co-simulation, simulating the RTL against a set of inputs. In the following sections, BM denotes the baseline model, BP-DSP denotes the DSP-optimized model, BP-MD denotes the DSP- and BRAM-optimized model, obtained from the multi-dimensional pruning described in Section III-A. The pruned models are accelerated using the code generation framework, described in Section III-C and the baseline model is accelerated using the built-in matrix multiplication method, described in Algorithm 1.

### _DSP-aware pruning_

First, we consider pruning all the weights processed by the same DSP block, by varying the RF. Results for jet and SVHN classification are reported in Tables II and III, respectively. In jet classification, the proposed method achieves a mean reduction of 9.4x and 3.1x in DSP and BRAM utilization, respectively. In SVHN classification, the proposed method achieves a mean reduction of 3.2x and 1.4x in DSP and BRAM utilization, respectively. In both cases, logic utilization is reduced as well. By increasing the RF, a smaller reduction in DSP utilization is obtained - due to the scope of pruning, with a higher RF, each pruning step removes more weights, leading to a higher drop in accuracy.

In jet classification, a noticeable drop in inference latency is observed, by switching from task-level pipelining to a system-level pipeline. For SVHN classification, the drop in latency is lower, as task-level pipelining must be enabled to ensure data is processed in a FIFO manner. However, through pruning, logic is reduced and the HLS compiler is able to schedule the RTL with a lower latency for the same clock frequency. Pruning can aid the closure of timing as well as the reduction of congestion, by reducing the overall LUT utilization. While DSP-aware pruning primarily targets weights processed by the same DSP, BRAM utilization is also reduced. For high sparsities, consecutive DSP blocks will be

Fig. 4: End-to-end flow of resource-aware pruning.
pruned, corresponding to one block of RAM. The reduction is lower for SVHN classification, since activation tensors are stored in BRAM, and, DSP-aware pruning has no effect on activation tensors.

### _Multi-dimensional pruning_

Next, we consider multi-dimensional pruning, by pruning all the weights in a single block of RAM, corresponding to several DSP blocks. To aid pruning, the models were quantized using 18-bit precision, instead of the default 16-bit precision. The results for jet classification are reported in Table II, with reductions ranging between 2.3x and 5.2x in BRAM utilization and 3.9x and 11.6x in DSP utilization. Compared to DSP-aware pruning, multi-dimensional pruning achieves higher reductions in block memory, with lower reductions in DSP utilization. Similar to DSP-aware pruning, LUT and FF utilization decreased in all cases.

### _Heterogeneous pruning_

Finally, we consider the example of heterogeneous, multi-dimensional pruning for fast image classification with a LeNet-like architecture. The LeNet model has significantly more parameters than the models used for jet and SVHN classification. To ensure low latency and meet Vivado partitioning and unrolling limits, the hardware configuration of each layer is tuned manually, according to its size and complexity. Table IV summarizes the strategy and RF for every layer, identified by its type (CONV, FC) and position in the network.

CONV layers compute the output for one pixel at a time, performing fewer multiplication over a longer period of time. For a CONV layer with output width \(W\), height \(H\) and reuse factor \(R\), the latency is approximately \(HWR\). Furthermore, CONV layers have a small number of weights, which can be stored in registers (_Latency_ strategy), further reducing the latency, while meeting the Vivado partition limits. Therefore, CONV layers are implemented using _Latency_ strategy with RF=1, performing all multiplications in parallel per output pixel. Since each weight maps to one DSP block and BRAM is not used for storing weights, unstructured pruning can be used to reduce resource utilization in CONV layers.

FC layers have a large number of weights, and, as such cannot perform all multiplications in parallel. However, the latency of FC layers is low and approximately equal to the RF. Furthermore, storing weights in registers would exceed the Vivado partition limit. Therefore, _Resource_ strategy, storing weights in BRAM, with the smallest possible RF passing synthesis, is chosen for FC layers. FC layers are quantized to 18 bits and pruned with resource-aware structures so that each pruned structure saves one block of RAM and two DSP blocks.

The resource utilization \([\text{DSP, BRAM}]\) of individual weights in CONV layers is \([1,0]\) and \([2,1]\) for resource-aware structures in FC layers. This example showcases the power of the knapsack formulation: different layers will have different
resource utilization per target structure and varying contributions to network accuracy. The results of multi-dimensional pruning are summarized in Table V. With a clock period of 10ns, there is a noticeable reduction in DSP utilization, with a slight reduction in BRAM utilization. However, increasing the clock period will drive a further reduction in BRAM, linked to duplication of logic by Vivado to close timing for a clock period of 10ns. In both cases, the inference was completed in less than 10\(\upmu\)s. Finally, to show the true power of pruning, we accelerate the pruned network on a mid-range accelerator card, Zynq UltraScale+ MPSoC ZCU102 (FPGA part xczu9eg-ffvb1156-2-e), intended for automotive, industrial and communication applications [42]. While the baseline model (4,175 DSP) is too large to be accelerated with the target device (2,520 DSP), the pruned model was accelerated while using less than a third of the available resources. Through pruning, neural networks can be accelerated on lower-end FPGAs, as well as enable the acceleration of several, concurrent neural networks on the same FPGA.

## V Conclusions

We propose a novel pruning method, modeled using the knapsack problem with resource-aware tensor structures, achieving reductions up to 92% in DSP utilization and up to 81% reduction in BRAM utilization on particle physics benchmarks and fast image classification. Furthermore, while primarily targeting DSP and BRAM reduction, the proposed method has been shown effective in reducing inference latency and logic utilization. The proposed methods have been open-sourced, building on top of hls4ml, enabling long-term contributions as well as applications in other fields. A possible direction for future work is the integration of resource-aware pruning with quantization-aware training. Recent work [43, 44, 45] aimed to combine the two methods, maximizing resource savings.

## VI Acknowledgments

This work is supported by Engineering and Physical Sciences Research Council (EPSRC) grant EP/S030069/1 and NSF Institute A3D3, NSF 2117997.
