Design and Optimization of Residual Neural Network Accelerators for
  Low-Power FPGAs Using High-Level Synthesis
Residual neural networks are widely used in computer vision tasks. They
enable the construction of deeper and more accurate models by mitigating the
vanishing gradient problem. Their main innovation is the residual block which
allows the output of one layer to bypass one or more intermediate layers and be
added to the output of a later layer. Their complex structure and the buffering
required by the residual block make them difficult to implement on
resource-constrained platforms. We present a novel design flow for implementing
deep learning models for field programmable gate arrays optimized for ResNets,
using a strategy to reduce their buffering overhead to obtain a
resource-efficient implementation of the residual layer. Our high-level
synthesis (HLS)-based flow encompasses a thorough set of design principles and
optimization strategies, exploiting in novel ways standard techniques such as
temporal reuse and loop merging to efficiently map ResNet models, and
potentially other skip connection-based NN architectures, into FPGA. The models
are quantized to 8-bit integers for both weights and activations, 16-bit for
biases, and 32-bit for accumulations. The experimental results are obtained on
the CIFAR-10 dataset using ResNet8 and ResNet20 implemented with Xilinx FPGAs
using HLS on the Ultra96-V2 and Kria KV260 boards. Compared to the
state-of-the-art on the Kria KV260 board, our ResNet20 implementation achieves
2.88X speedup with 0.5% higher accuracy of 91.3%, while ResNet8 accuracy
improves by 2.8% to 88.7%. The throughputs of ResNet8 and ResNet20 are 12971
FPS and 3254 FPS on the Ultra96 board, and 30153 FPS and 7601 FPS on the Kria
KV26, respectively. They Pareto-dominate state-of-the-art solutions concerning
accuracy, throughput, and energy.
Filippo Minnella, Teodoro Urso, Mihai T. Lazarescu, Luciano Lavagno
Design and Optimization of Residual Neural Network Accelerators for Low-Power FPGAs Using High-Level Synthesis

###### Abstract

Residual neural networks (ResNets) are widely used in computer vision tasks. They enable the construction of deeper and more accurate models by mitigating the vanishing gradient problem. Their main innovation is the _residual block_ which allows the output of one layer to bypass one or more intermediate layers and be added to the output of a later layer. Their complex structure and the buffering required by the residual block makes them difficult to implement on resource-constrained platforms. We present a novel design flow for implementing deep learning models for field-programmable gate arrays (FPGAs) optimized for ResNets, using a strategy to reduce their buffering overhead to obtain a resource-efficient implementation of the residual layer. The current implementations of residual networks suffer from diminished performance and heightened computational latency attributable to the way residual blocks are implemented. Our high-level synthesis based flow encompasses a thorough set of design principles and optimization strategies, exploiting in novel ways standard techniques such as _temporal reuse_ and _loop merging_ to efficiently map ResNet models, and potentially other skip connection-based NN architectures, into FPGA. The models are quantized to 8-bit integers for both weights and activations, 16 bits for biases, and 32 bits for accumulations. The experimental results are obtained on the CIFAR-10 dataset using ResNet8 and ResNet20 implemented with Xilinx FPGAs using HLS on the Ultra96-V2 and Kria KV260 boards. Compared to the state-of-the-art on the Kria KV260 board, our ResNet20 implementation achieves \(2.88\times\) speedup with \(0.5\,\mathrm{\char 37}\) higher accuracy of \(91.3\,\mathrm{\char 37}\), while ResNet8 accuracy improves by \(2.8\,\mathrm{\char 37}\) to \(88.7\,\mathrm{\char 37}\). The throughputs of ResNet8 and ResNet20 are \(12\,971\,\mathrm{FPS}\) and \(3254\,\mathrm{FPS}\) on the Ultra96 board, and \(30\,153\,\mathrm{FPS}\) and \(7601\,\mathrm{FPS}\) on the Kria KV26, respectively. They Pareto-dominate state-of-the-art solutions with respect to accuracy, throughput, and energy.

## 1 Introduction

Convolutional neural networks (CNNs) have consistently achieved state-of-the-art results in many tasks, including computer vision and speech recognition [20]. Their success is based on high accuracy and performance due to the improved computational intensity of convolutional layers compared to previous approaches, requiring less memory bandwidth than fully connected (FC) layers [25]. The choice of hardware for implementing convolutional layers profoundly impacts their applicability. Central processing units (CPUs) are versatile and easy to program, but their architecture makes them relatively inefficient. Graphical processing units (GPUs) are designed to handle massive parallelism, allowing them to process multiple computations simultaneously. This aligns well with the inherently parallel nature of CNNs but their energy consumption is notably higher [6]. application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs) offer different tradeoffs of cost and flexibility for algorithm acceleration [40]. The latter are less performance and energy efficient due to their reprogrammability, but they have much lower design cost and can be more easily customized for a specific application. Neural networks (NNs) optimized for embedded applications [16, 41] are designed to run efficiently on devices with limited processing power, memory, and energy. They can perform very well on small datasets, such as CIFAR-10 [19] and MNIST [5], and are often used in real-time contexts where response timeliness and low latency are critical.

Residual neural networks (ResNets) [15] use residual blocks (see Fig. 1) to mitigate the vanishing gradient problem for deep networks through _skip connections_. They allow intermediate feature maps to be reprocessed at different points in the network computation, increasing accuracy. However, state-of-the-art implementations of _skip connections_ require significant on-chip buffering resources, which significantly reduce the benefits of streaming-based FPGA implementations. Thus, recent work has focused on optimizing and
shrinking the residual structure of NNs [33] and on finding quantization strategies that improve the efficiency of their hardware design [8].

Deep networks have many parameters and require extensive quantization to reduce their size to fit into the FPGA on-chip memory [22]. For this reason, widely used tools such as FINN [3] focus on low-bit quantization (such as 1-bit [28] or 2-bit [4]) and make suboptimal use of resources for higher-bit quantization, as we will show later. However, low-bit quantizations degrade NN accuracy and may not be suitable for accurate inference for complex problems [30]. We propose an efficient residual block architecture for a concurrent dataflow FPGA implementation, in which each layer is a process and activations are streamed between processes, with the following main contributions:

* An optimized architecture for CNNs that supports residual networks and minimizes buffering resources, allowing on-chip storage of the parameters and activations using 8 bits, which have been shown to achieve good accuracy for our target NN architectures [21].
* A custom high-level synthesis (HLS) code generation flow from the Python model to the FPGA bitstream using Vitis HLS [37].
* The validation of the architecture and the implementation flow using the ResNet8 and ResNet20 residual NNs on CIFAR-10 targeting the Ultra96-V2 and Kria KV260 FPGA boards, to demonstrate the advantages of the proposed solution.

The rest of the paper is organized as follows. Section 2 presents the background and motivation for this work. Section 3 discusses training and quantization, and describes the accelerator architecture with a special focus on skip connection management. Section 4 presents the experimental setup and discusses the results. Section 5 concludes the paper.

## 2 Related Work

The field of FPGA-based acceleration for deep neural networks has gained significant attention due to its potential to achieve high-performance and energy-efficient inference. Several approaches and architectures have been proposed in the literature to address this challenge [10]. In systolic array overlay-based architectures, each processing element (PE) is a single instruction multiple data (SIMD) vector accumulation module, which receives activation inputs and weights in each cycle from the horizontally and vertically adjacent PEs. Pipelined groups of PEs with short local communication and regular architectures can achieve high clock frequencies and efficient global data transfers [32]. The overlay architecture [39] performs the computation of the convolution layers in sequence over a systolic array. However, despite its flexibility, it has high latency due to frequent transfers between external memory [double data rate (DDR) or high bandwidth memory (HBM)] and on-chip memory.

An alternative approach is to implement a _custom dataflow architecture where each layer is associated with a compute unit_. This structure can be pipelined, and activations and weights can be stored in on-chip memory (OCM), reducing latency and increasing throughput. The main limitation of this type of architecture is the number of digital signal processor blocks (DSPs) and lookup tables (LUTs) required to implement the convolutional layers, as well as the size of on-chip buffers for weight storage [27], while activations are streamed from layer to layer. Since streaming tasks have well-defined pipelining rates with static data flows, a customized approach can lead to optimized processing, resulting in improved performance and resource saving.

Widely recognized as one of the leading frameworks for deploying deep neural networks (DNNs) on FPGAs, Xilinx Vitis AI [1] provides a comprehensive set of tools specifically designed for optimizing and deploying DNNs on Xilinx FPGAs. With support for popular frameworks such as TensorFlow [7], PyTorch [11], and Caffe [9], it incorporates various optimization techniques such as pruning, quantization, and kernel fusion to improve performance and reduce memory consumption. The deep learning processor unit (DPU) is the accelerator core used in Vitis AI and consists of several key modules, including a high-performance scheduler module, a hybrid computing array module, an instruction fetch unit module, and a global memory pool module [36]. The DPU is responsible for executing the microcode of the spec

Figure 1: Basic residual block with a long branch with two convolutional layers, and the skip connection (red branch) that must store its input activations until output activation is generated, requiring much memory.
ified DNN model, known as the _xmodel_. Vitis AI uses an overlay-based architecture where model weights and biases are stored in DDR memory and cached in the on-chip weight buffer during inference. Input and output data of the PE array are also cached in OCM. This architecture scales very well for DNN, but may have higher resource utilization and less performance compared to custom dataflow accelerators due to its general-purpose nature and the overhead associated with off-chip memory accesses.

Another widely used tool is FINN [3], an open source framework developed by Xilinx that allows the generation of highly optimized DNNs for FPGA acceleration, with emphasis on dataflow-style architectures. FINN uses HLS to convert trained DNN models into hardware intellectual property (IP) blocks that can be easily integrated into FPGA-based systems. While FINN offers significant customization capabilities, it is primarily designed for low-bitwidth quantization schemes, such as binarized networks. Achieving high performance with FINN often leads to lower accuracy and/or higher resources, particularly when using the 8-bit quantization that has been shown to be the best compromise between resources and accuracy.

[31] evaluates the accuracy, power, throughput, and design time of three different CNNs implemented on FPGAs and compares these metrics to their GPU equivalents. A comparison was also made between a custom implementation of two DNNs using System Verilog and an implementation using the Xilinx tools FINN and Vitis AI [24]. In addition, [14] reports a comparison between FINN and Vitis AI using a widely used set of ResNet model configurations.

We propose an optimized pipelined dataflow architecture tailored for better resource management. Our solution specifically targets residuals NNs and allows using more bits during quantization than e.g. FINN, to improve the trade-off between accuracy, throughput, and memory usage [35]. Its effectiveness is compared with Vitis AI, FINN, and custom resource-efficient approaches [42], demonstrating its potential to efficiently implement complex residual NNs with state-of-the-art performance in terms of latency, energy, accuracy, and resource consumption.

## 3 Methodology

Fig. 2 shows the high-level view of the flow that we use to generate C++ code from the quantized NN model, which includes the following main steps:

* Use Brevitas [26] for NN quantization and extract its graph in QONNX format, which provides an easy-to-parse description of the network, including information such as layer type, input and output quantization, and layer connections (see Section 3.1);
* Generate the C++ of the top function that instantiates all the layers of the network (see Section 3.7 and Section 3.2);
* Generate the register-transfer level (RTL) code using Vitis HLS and a set of model-independent synthesizable C++ libraries that we wrote to implement the optimized layers (see Section 3.3);
* Import the RTL code as a block into a Vivado design and generate the bitstream for FPGA programming.

### Quantization

Quantization is done using the Brevitas framework [26]. NNs are trained using PyTorch, and both the weights (\(\mathit{bw_{\text{w}}}\)) and the activations (\(\mathit{bw_{\text{x}}}\)) are represented as 8-bit integers, because for a variety of applications this is a good compromise between memory usage and accuracy ([21]), while the biases are represented as 16-bit integers (\(\mathit{bs_{\text{b}}}\)) for the same reason.

NN training uses floating-point calculations and models quantization by clamping and rounding. Back-propagation uses dequantized inputs and weights to improve convergence and accuracy, while loss evaluation uses quantization to match the results of the hardware implementation. Inference on the FPGA uses multiplications of operands quantized to different sizes, while the results are accumulated in 32-bit registers to avoid overflows and efficiently map to FPGA resources such as DSPs, as discussed in Section 3.3.

Figure 2: Implementation flow
The quantization \(Q(\cdot)\) of a value \(b\) on \(bw\) bits \[a=Q(b)=\text{clip}(\text{round}(b\cdot 2^{bw-s}),a_{\text{min}},a_{\text{max}}) \cdot 2^{s}\quad s\in\mathbb{N}\] (1) \[a_{\text{min}}=\mathit{act}_{\text{min}}(s)=\left\{\begin{array}{ll}0& \text{if unsigned}\\ -2^{bw-1-s}&\text{if signed}\end{array}\right.\] (2) \[a_{\text{max}}=\mathit{act}_{\text{max}}(s)=\left\{\begin{array}{ll}2^{ bw-s}-1&\text{if unsigned}\\ 2^{bw-1-s}&\text{if signed}\end{array}\right.\] (3)

uses \(s\) as scaling factor, \(a_{\text{min}}\) is the lower clipping bound and \(a_{\text{max}}\) is the higher one. All _zero points_ are set to zero and omitted in the expressions above, while the _scaling factors_ are set to powers of two to map alignment operations between weights and activations into hardware-friendly bit shifts [18].

The bias scaling factor \(s_{\text{b}}\) is calculated as the sum of the input scaling factor \(s_{\text{x}}\) and the weight scaling factor \(s_{\text{w}}\).

After the training, to avoid the hardware implementation of floating point operations, the batch normalization layers are merged with the quantized convolution layers [17] and retrained to calibrate and tune the quantization parameters. The final model is exported to the QONNX format [12, 29].

### Accelerator architecture

The _code generation_ step in Fig. 2 works on the optimized QONNX graph, i.e. after ReLU and batch normalization were merged with convolutional layers, and provides a C++ _top function_ that instantiates all the tasks (also known as dataflow processes) needed to implement network inference:

* _Computation task_: one for each convolution or pooling node to implement the layer computations (see Section 3.3).
* _Parameter task_: one for each convolution to correctly provide the convolution parameters to the computation task. One additional task is added to load parameters from _off-chip memory_ if UltraRAM (URAM) storage is used (see Section 3.4).
* _Window buffer tasks_: multiple tasks for each convolution or pooling node, to format the input data to the computation tasks (see Section 3.6).

All tasks are coded in a reusable, templated C++ library that can adapt to different activation and weight tensor dimensions, data types, and computational parallelism. Each task is composed of a main loop that performs multiple operations.

To increase the accelerator throughput, pipelining is enabled at two different levels:

* Inter-task: concurrent layer execution is achieved using the _dataflow_ pragma in the body of the top function. There is one computation task and, possibly, multiple window buffer tasks running for each of the network layers. The latency of the slowest task determines the overall accelerator throughput.
* Intra-task: concurrent operation execution is used to reduce and balance task latencies. _Computation tasks_ are the most computationally intensive. Thus each top loop inside them is partially unrolled by a factor computed at compile time and based on the complexity of the corresponding computation. This effectively allocates a number of PEs, one for each unrolled iteration, for each _computation task_. Each PE performs one or more multiply and accumulates (MACs) operations per clock cycle. See Section 3.3 and Section 3.5 about how low-level DSP packing is used to increase the number of MACs executed by a DSP unit in a clock cycle. If the _computation task_ belongs to a convolution, the related _parameter task_ main loop is unrolled by the same factor, to provide enough data to support the increased computations parallelism.

An integer linear programming (ILP) model described in Section 3.5 is used to globally optimize the unroll factors (number of PEs) that maximizes NN throughput under DSP resource constraints (DSPs are the most critical FPGA resource for the NN architectures that we considered in this paper).

Network inputs and outputs are implemented as _streams_ of data. direct memory access (DMA) blocks read/write input/output tensors to/from the off-chip memory. Streams also transfer data between tasks in order minimize the memory requirements for on-chip activations storage. (See Section 3.6)

The _data-driven_ execution approach is chosen to process the frames sequentially and as a continuous stream. This is achieved in Vitis HLS by using the ap_ctrl_none pragma in the top function that models the entire NN. Each task is then operating as soon as input data are available.

Inference begins as soon as the DMA attached to the input port of the top-level interface is enabled to download input images. Each task is pipelined; the first stage reads the input stream, while the others process the data and write the output stream. As a further, tool-specific, implementation detail, intra-task pipelines are not flushable, which would consume more resources, but stalling with auto-rewind disabled, to both save resources and avoid deadlocks. Auto-rewind would start a new loop execution while the pipeline is processing the last iterations of the old one, but with data-driven ap_ctrl_none dataflow it would cause deadlocks at runtime. Performance is largely unaffected because the _intra-task pipeline_ latency is very small, just a few cycles, compared to the task latency, which is proportional to the number of
iterations of the _intra-task pipeline_.

### Convolution computation task

Each convolution _computation task_ receives a window of input activations from a _window buffer task_. Fig. 4 shows the pseudo-code for the convolution computation and examples of how the computation pipeline receives input data and computes the partial results. The PARFOR pseudo-code is implemented as an unrolled loop in synthesizable C++.

Input tensors are mostly provided in depth-first-order to each convolution \(i\), as discussed below. The innermost loops are completely unrolled over the filter dimensions (\(\mathit{fh}_{i},\mathit{fw}_{i}\)) and part of the output channel (\(\mathit{och}_{i}\)) dimension. This unroll factor \(\mathit{och}_{i}^{\mathit{par}}\), where "par" means that the execution will be fully data-parallel, defines the number of PEs, as discussed above. It is chosen by the algorithm described in Section 3.5. The \(\mathit{och}_{i}^{\mathit{par}}\) unroll factor is limited by on-chip memory bandwidth and the number of arithmetic units that can be used simultaneously. Increasing the number of output channels computed in parallel per clock cycle requires the corresponding filter parameters to be provided in parallel, i.e., higher memory bandwidth and potentially more BlockRAM (BRAM) resources.

Another optimization changes the order in which the windows are given to the data path, instead of channel first order, and unrolls along the output tensor width (\(\mathit{ow}_{i}\)) loop by a factor (\(\mathit{ow}_{i}^{\mathit{par}}\)). Unrolling along the tensor width allows us to reduce the computation time without requiring more memory bandwidth for the filter parameters, at the cost of more partitioning of the input activation window buffer, and hence of potentially more BRAM resources.

This also allows the weights to be reused within an output stationary dataflow and can be exploited in future work where larger networks are considered and the off-chip memory is used to store network parameters.

We now discuss how we exploit the DSP packing method described in [38] to reduce the hardware overhead of computing quantized data, by performing multiple operations on a single DSPs block. Unlike \(\mathit{och}_{i}^{\mathit{par}}\), which is resource dependent, \(\mathit{ow}_{i}^{\mathit{par}}\) depends on the activation quantization bits. Even though the number of operations packed in a DSP depends on the number of bits, this work only used the configuration described in [38], which presents a method for \(\mathit{bw}_{i}=8\) for both parameters and activations.

Fig. 5 shows two examples of calculation pipelines, with different values of \(\mathit{ow}_{i}^{\mathit{par}}\). Each gray box is a PE that receives:

* _Input activations:_\(\mathit{ow}_{i}^{\mathit{par}}\) inputs. These values change at each iteration of the \(\mathit{och}_{i}^{\mathit{groups}}\) loop. The input activations are multiplied in parallel by the PE input weight and are provided by the corresponding _window buffer tasks_.
* _Input weight_: one input. This value is updated at each clock cycle. The input weight is provided by the corresponding _parameter task_.

\begin{table}
\begin{tabular}{l l} \hline \hline Symbol & Description \\ \hline \(\mathit{ich}_{i}\) & Input tensor channels \\ \(\mathit{ih}_{i}\) & Input tensor height \\ \(\mathit{iw}_{i}\) & Input tensor width \\ \(\mathit{och}_{i}\) & Output tensor channels \\ \(\mathit{oh}_{i}\) & Output tensor height \\ \(\mathit{ow}_{i}\) & Output tensor width \\ \(\mathit{fh}_{i}\) & Filter tensor height \\ \(\mathit{fw}_{i}\) & Filter tensor width \\ \(\mathit{s}_{i}\) & Convolution stride \\ \hline \hline \end{tabular}
\end{table}
Table 1: Symbol definitions for layer \(i\)

Figure 3: Accelerator architecture with direct memory access (DMA) blocks for memory transfers (grey box) and concurrent tasks [computation (yellow), buffer (red), and parameter (green)] communicating through data streams. Parameter loading from off-chip memory to URAMs (dashed) can be enabled on platforms supporting it.
* _Partial accumulation_: one input. This value is updated at each clock cycle. The partial accumulation is provided by the previous pipeline stage.

Each PE receives an input weight every clock cycle, so sufficient OCM bandwidth must be provided (see Section 3.4).

The two pipelines in Fig. 5 highlight how \(\mathit{och}_{i}^{\mathit{par}}\) allocates multiple PEs per pipeline stage (horizontal unroll), and how \(\mathit{ow}_{i}^{\mathit{par}}=2\) modifies the mapping of the input activations to the different stages of the pipelines, thus increasing the number of computations for each PE. The partial accumulation entering each PE comes from the previous pipeline stage. The only exception is the first stage, which receives as value to accumulate the _bias_ of the convolution.

Each MAC calculation, for the case \(\mathit{ow}_{i}^{\mathit{par}}=1\), is done using a DSP from the _Xilinx_ architecture. If \(\mathit{ow}_{i}^{\mathit{par}}=2\) the two MACs have reduced resource usage thanks to the technique described in [38].

As shown by the pipeline in Fig. 5 with \(\mathit{ow}_{i}^{\mathit{par}}=2\), the operation packing is done by multiplying 2 activations (\(A\), \(D\)) by 1 parameter (\(B\)) and accumulating to a partial result (\(P_{i-1}\)). The output (\(P_{i}\)) is passed to the next pipeline stage.

The multiplier of the DSPs receives one \(27\,\mathrm{bit}\) and one \(18\,\mathrm{bit}\) input. The former packs the two activations:

while the latter contains the sign-extended parameter:

Figure 4: Convolution architecture: The data flow is output stationary, and for \(\mathit{och}_{i}^{\mathit{par}}\) output channels the contribution of the input window \(\mathit{fh}_{i},\mathit{fw}_{i}\) is evaluated every clock cycle. Data is written to the output after all input channels have been processed. The dataflow setup for \(\mathit{ow}_{i}^{\mathit{par}}=1\) and \(\mathit{ow}_{i}^{\mathit{par}}=2\) is shown in the two schematics. The input activations are loaded simultaneously, along orange lines, into each gray box, which is a PE. PEs performs a _MAC_ operation and the partial results move through the pipeline from top to bottom along the green lines.
The two operands are multiplied into a \(36\,\mathrm{bit}\) word (\(M\)):

The DSP also adds the partial product to the accumulation coming from the previous pipeline stage, \(P_{i-1}\):

At the end of the chain, a restore stage ensures that the \(p_{v}\) sign does not create errors in the final result:

Note that in our specific use case, where activations and parameters are quantized on \(8\,\mathrm{bits}\), we can at most chain **7** packed DSPs, because of the limited padding between the two words, namely \(2\,\mathrm{bits}\), and the restore mechanism which corrects \(1\,\mathrm{bit}\) overflow. However, for convolution filters with \(\textit{fn}_{i}=3\) and \(\textit{fw}_{i}=3\), the DSP chain should have a length of 9. Hence we split the chain into \(2\) subparts that respect the maximum length condition. The partial accumulations coming from the different chains are then added together in an additional stage, and the result coming from the DSPs pipeline is finally added to the accumulation register.

Registers keeping the partial results of the MAC between multiple iterations are sized in order to avoid overflows in the computations. Considering models with \(\textit{bw}_{i}\) bit quantization, the accumulated values from the product between an activation and a parameter, have a width equal to \(2\cdot\textit{bw}_{i}\), using same bit-width for activation and parameter quantization. For each convolution, the number of accumulations (\(N_{i}^{acc}\)) performed to compute each output value is

\[N_{i}^{acc}=\textit{och}_{i}\cdot\textit{ich}_{i}\cdot\textit{fh}_{i}\cdot \textit{fw}_{i}. \tag{4}\]

Since the addend has \(2\textit{bw}_{i}\) bits, the final accumulation register must have a width equal to

\[\textit{bw}_{i}^{acc}=\lceil\log_{2}(N_{i}^{acc})\rceil+2\textit{bw}_{i}. \tag{5}\]

Considering the worst case for _Resnet8_ and _Resnet20_ with \(8\,\mathrm{bit}\) quantization, the required bitwidth is

\[N_{i}^{acc}=32\cdot 32\cdot 3\cdot 3=9216 \tag{6}\]

\[\textit{bw}_{i}^{acc}=\lceil\log_{2}9216\rceil+2\cdot 8=14+16=30. \tag{7}\]

The accumulation register size is chosen to be \(32\,\mathrm{bit}\) because it ensures no overflow, and using standard C++ types improves C simulation speed.

### Parameter task

Each convolution layer of the QONNX network graph has a _parameter task_ in the top function, feeding the computation pipeline with data from on-chip memory. Depending on the target FPGA, parameters may be stored in:

* _BRAMs_: they can store up to \(4\,\mathrm{KB}\) each and can be initialized by the bitstream. The parameters for each convolution are stored in separate arrays, one for each weight and bias of the convolutions, because each is accessed by a specific _parameter task_.
* _UltraRAMs (URAMs)_: they can store \(32\,\mathrm{KB}\) of data each (allowing higher performance) but require dedicated hardware for initialization (a DMA-driven input stream). The parameters for each convolution are packed into a single array stored in off-chip DRAM (also accessible by the host) and transferred by DMA once at power-up. A concurrent task in the dataflow architecture splits and distributes the input parameter stream to the tasks that handles the parameters of each convolution. Each _Parameter task_ provides the filter data to the computation pipeline and caches it in URAMs at the first iteration for reuse (hence the next URAMs accesses are read-only).

The Ultra96 board lacks URAM, so BRAM is used. The KRIA KV260 board uses the URAM.

As discussed in Section 3.3, the main loop of each convolution's _computation task_ consumes \(\textit{cw}_{i}=\textit{och}_{i}^{par}\cdot\textit{fh}_{i}\cdot\textit{fw}_{i}\) filter data per clock cycle. The \(\textit{ow}_{i}^{par}\) unroll factor does not contribute because each parameter is used for multiplication with \(\textit{ow}_{i}^{par}\) activations. To avoid stalling the computation pipeline, the _parameter task_ must write \(\textit{cw}_{i}\) weights every clock cycle and read the same amount of data from the BRAMs or URAMs. Arrays are then reshaped by a factor equal to \(\textit{cw}_{i}\), using the array_reshape pragma, to achieve the required memory bandwidth.

### Throughput optimization

To avoid stalling, all streams are sized appropriately by our configuration _Python_ script based on their type, as follows.

Streams created by _parameter tasks_ supply _computation tasks_ with a token size equal to the computational parallelism of the consuming convolution, \(\textit{och}_{i}^{par}\), every clock cycle. Since the producer and consumer write and read one token per clock cycle, the stream size is 2.

The sizes of the streams produced by _window buffer tasks_ are discussed in Section 3.6.

The output stream from _computation tasks_ must consider \(\textit{och}_{i}^{par}\) and \(\textit{ow}_{i}^{par}\). The pseudocode in Fig. 4 shows that
computation tasks_ write a burst of \(\mathit{och}_{i}\cdot\mathit{ow}_{i}^{par}\) output activations, grouped into tokens of size \(\mathit{och}_{i}^{par}\) to not stall the pipeline. When packing is applied, the output stream is split into \(\mathit{ow}_{i}^{par}\) parallel channels to ensure enough bandwidth. Each channel is implemented by a first in first out (FIFO) of size \(\mathit{och}_{i}^{groups}=\mathit{och}_{i}/\mathit{och}_{i}^{par}\) to store the burst transactions completely.

As mentioned above, using the _dataflow_ paradigm and assuming optimal stream sizing to avoid stalling, accelerator throughput is limited by the slowest concurrent process. Therefore, the throughput \(\mathit{Th}\) of each layer unit must be balanced for optimal performance. The latency of each module depends on the number of computations for each input frame \(c\) and the computational parallelism \(\mathit{cp}\) required for each block \(i\). The number of computations for a convolutional layer is

\[c_{i}=\mathit{oh}_{i}\cdot\mathit{ow}_{i}\cdot\mathit{och}_{i}\cdot\mathit{ ich}_{i}\cdot\mathit{fh}_{i}\cdot\mathit{fw}_{i}. \tag{8}\]

Since the parameter \(c_{i}\) is fixed and depends on the chosen network architecture, the throughput per layer is set by the number of compute units allocated to each _computation task_ implementing a layer. As shown in the pseudocode in Fig. 4, computation parallelism \(\mathit{cp}_{i}\) is

\[\mathit{cp}_{i} =k_{i}\cdot\mathit{och}_{i}^{par}\cdot\mathit{ow}_{i}^{par}, \tag{9}\] \[\text{with}\quad k_{i} =\mathit{fh}_{i}\cdot\mathit{fw}_{i},\mathit{och}_{i}^{par}, \mathit{ow}_{i}^{par}\in\mathbb{N}. \tag{10}\]

Since the filter size \(k_{i}\) is defined by the model and \(\mathit{ow}_{i}^{par}=2\), because for this work we consider all the quantization bit-widths equal to \(8\,\mathrm{bit}\), the variable to optimize is \(\mathit{och}_{i}^{par}\), i.e. \(\mathit{cp}_{i}\) is an integer multiple of the filter size.

The throughput of each task, \(\mathit{Th}_{i}\) frame per second (FPS), depends on the variable \(\mathit{och}_{i}^{par}\)

\[\mathit{Th}_{i}=\mathit{Th}\left(\mathit{och}_{i}^{par}\right)=\frac{\mathit{ cp}_{i}}{c_{i}}=\frac{k_{i}\cdot\mathit{och}_{i}^{par}\cdot\mathit{ow}_{i}^{par}}{c_{i}}. \tag{11}\]

Considering a network with \(N\) convolutional layers, Algorithm 1 shows an ILP formulation of throughput optimization. If \(i_{\max}\in[1,N]\) is the index of the layer with the highest \(c_{i}\), then the goal is to balance the throughput of all layers

\[\forall i\in[1,N]\quad\mathit{Th}\left(\mathit{och}_{i_{\max}}^{par}\right)= \mathit{Th}\left(x_{i}\right)\implies\mathit{cp}_{i}=\mathit{cp}_{i_{\max}}r_ {i} \tag{14}\]

with \(r_{i}=c_{i}/c_{i_{\max}}\). Then the number of resources needed for each layer can be calculated, given the resources allocated for layer \(i_{\max}\). The total number of parallel computations allocated is

\[\mathit{cp}_{\mathrm{tot}}=\sum_{i=1}^{N}\mathit{cp}_{i}=\sum_{i=1}^{N} \mathit{cp}_{i_{\max}}r_{i}. \tag{15}\]

From (13), \(N_{\mathrm{PAR}}\) limits the maximum number of computations that can be done in parallel and depends on the platform. The FPGAs on the Ultra96 and KRIA KV260 boards that we are considering have \(360\) and \(1248\) DSPs, respectively. During hardware generation, \(N_{\mathrm{PAR}}\) is set to the number of DSPs on the target board.

The ILP can then maximize the throughput of the network by optimizing the parameters for the \(i_{\max}\) layer (12) and automatically configuring the template parameters of the tasks.

### Window generation

Given a convolution input tensor, we only need to store on-chip enough data to provide the input window to the _intra-task_ pipeline of each computational task. For example, Fig. 6 shows an input tensor and the input window mapping for a convolution with \(\mathit{fh}_{i}=3,\mathit{fw}_{i}=3\). It is important to highlight that the activations are produced using _depth-first_ order by the convolution that creates the input tensor (Fig. 4), while the input window is distributed over one channel and multiple lines. It is thus necessary to store all the lines needed to generate an input window, so each window buffer (also called line buffer in the literature) should be sized to accommodate the required activations. The portion of the input tensor (\(B_{i}\)) that the buffer must retain to create an input window is highlighted in Fig. 6

\[B_{i}=[(\mathit{fh}_{i}-1)\cdot\mathit{iw}_{i}+\mathit{fw}_{i}-1]\cdot\mathit{ ich}_{i}. \tag{16}\]

This size is constant over time because each time that the buffer reads an activation and generates a window, it disc

Figure 6: Input window mapped on the input tensor, \(\mathit{ow}^{par}{}_{i}=1\). Fig. 4 shows how the window elements map to the computation pipeline.
The _window buffer tasks_ retrieve from the input buffer the \(B_{i}\) activations required for the windows. At the maximum unroll factor, \(\mathit{och}_{i}^{\mathit{par}}=\mathit{och}_{i}\), each intra-task pipeline of the _computation task_ processes one input window per clock cycle.

The data read by the _window buffer tasks_ from the input activation buffer is \(\mathit{fh}_{i}\cdot\mathit{fw}_{i}\), i.e. a convolution window. The data needed for the input window is not contiguous and cannot be read by directly addressing the buffer, because it is stored sequentially in a FIFO with only one read port available. To provide the necessary bandwidth, the FIFO must be partitioned into \(\mathit{fh}_{i}\cdot\mathit{fw}_{i}\) parts, connected sequentially as shown in Fig. 7. Optimizing the window buffer to reduce the required partitioning in cases that allow a lower window generation rate is left for future work.

The size of each FIFO, \(S_{1}\), \(S_{2}\), is the distance, in number of activations, between successive values of the same input window, considering that the tensor is processed in _depth-first_ order. \(S_{1}\) represents the distance between two activations within the same row of the input window, and it is equal to the number of channels \(\mathit{ich}_{i}\) in the tensor. In contrast, \(S_{2}\) covers the gap between two activations in different rows of the input window, and it is directly proportional to one row (\(\mathit{ich}_{i}\cdot\mathit{iw}_{i}\)) in the input tensor, minus the filter width \(\mathit{fw}_{i}\). Each _task_\(T_{i}\) reads from a FIFO the data for an input window position, \(i\), and provides the input for the next FIFO slice, \(i+1\).

The _padding_ task, if enabled, reads at each cycle the data from _task_\(T_{i}\) for positions \(i\) that do not require padding, and replaces with \(0\) the positions of the input window that must be padded. Thanks to the concurrent execution and padding-aware control of the _window buffer tasks_ and _padding task_, the first buffer slices can be initialized with the tensor values of the next frame, while the last ones generate the final windows of the previous frame, avoiding the latency overhead caused by initializing the FIFOs.

The structure of the _window buffer_ depends on the unroll factor \(\mathit{ow}_{i}^{\mathit{par}}\). Fig. 8 shows the input window mapped to the input tensor with \(\mathit{ow}_{i}^{\mathit{par}}=2\), for which all considerations made before about \(\mathit{ow}_{i}^{\mathit{par}}=1\) apply.

The input buffer size is

\[B_{i}=[(\mathit{fh}_{i}-1)\cdot\mathit{iw}_{i}+\mathit{fw}_{i}]\cdot\mathit{ich }_{i} \tag{17}\]

so the overhead with respect to (17) is minimal. The buffer must be partitioned to ensure the required window production rate. With \(\mathit{ow}_{i}^{\mathit{par}}=2\), the elements of the input window are \((\mathit{fw}_{i}+\mathit{ow}_{i}^{\mathit{par}}-1)\cdot\mathit{fh}_{i}\). Fig. 9 shows how the buffer is partitioned according to the required bandwidth.

The main difference between Fig. 7 and Fig. 9 is how the activations flow in the different FIFO slices.

Given an input filter of size \(\mathit{fh}_{i}\cdot\mathit{fw}_{i}\), an input activation is multiplied, at different times, by a value in each position of the filter window.

If \(\mathit{ow}_{i}^{\mathit{par}}=1\), there is a one-to-one correspondence between the positions of the input window and those of the filter window. This means that the activation must pass through all FIFO slices, because each of them represents a position \(i\) of the input/filter windows.

If \(\mathit{ow}_{i}^{\mathit{par}}=2\), the input window keeps two windows that are multiplied by the parameter window, i.e. part of the activations are evaluated in two adjacent positions for each input window (\(i,i+1\)). Thus, the output of \(T_{i}\) must be connected to the input of the FIFO slice \(i+2\) to ensure correct data flow.

Figure 8: Input window mapped on the input tensor, \(\mathit{ow}_{i}^{\mathit{par}}=2\), retaining two computation windows (red and blue). Fig. 4 shows how the window elements map to the computation pipeline.

Figure 7: Buffer partitioning, \(\mathit{ow}_{i}^{\mathit{par}}=1\). Yellow boxes are the FIFOs with their sizes. Orange boxes are tasks that read and write the FIFOs. Padding is applied before generating the box for the convolution.
### Graph Optimization

The main contribution of this paper is to provide a structured methodology to efficiently implement a residual block in a dataflow accelerator with concurrent processes.

The same considerations from Section 3.6 can be extended to network graphs with multiple nodes processing the same input tensor, i.e. residual blocks, as shown in Fig. 10, to provide a more general solution. Considering a tensor processed by multiple convolutions, multiple branches start from the convolution that generates the tensor. In residual networks the branches are then merged by an _add_ layer.

Fig. 10 shows _Resnet20_ and _Resnet8_ residual block topologies with _2 branches_ per input tensor and _0 or 1 convolutions_ on skip connection (the branch crossing fewer convolutions).

The _add_ layer adds the values from the _2 branches_. Because of the dataflow architecture, the operation starts as soon as both input streams have data. However, the time required to fill each stream is different. The skip connection stream that reaches the _add_ node is filled in parallel with the _conv0_ input stream in the case without downsampling, or after \(ich_{i}\) cycles in the case with downsampling. The input stream from the long branch is filled as soon as _conv1_ provides its first output activation. As shown by Fig. 6, _conv1_ starts processing data as soon as its input buffer is full. The amount of data buffered for skip connections, \(B_{\text{sc}}\), is equal to the amount to be processed by _conv0_ and is sufficient to start _conv1_ operations. To calculate this value we use the _receptive field_[2], which is the portion of feature maps related to successive layers that contribute to produce an activation.

Fig. 11 shows the _receptive field_ of the _conv1_ window with respect to the _conv0_ that generates it. \(B_{\text{sc}}\) is the buffering of all receptive fields projected from the activation in the _conv1_ line buffer as soon as it starts computing. From [2], as shown in Fig. 11, the data to store for each receptive field \(B_{\text{r}}\) is

\[rh_{0} =fh_{1}+fh_{0}-1 \tag{18}\] \[rw_{0} =fw_{1}+fw_{0}-1\] (19) \[B_{r} =rh_{0}\cdot rw_{0}. \tag{20}\]

Sliding the receptive field window over \(ich_{i},iw_{i}\), the obtained buffering \(B_{\text{sc}}\) is

\[B_{\text{sc}}=[iw_{0}\left(rh_{0}-1\right)+rw_{0}]\,ich_{0}. \tag{21}\]

In the dataflow architecture used in the final implementation, the "bypass" branch must store its input activation data

Figure 11: Receptive field of the _conv1_ window. For clarity, the \(ich\) dimension is omitted and _conv1_ stride is assumed to be 1 (\(s_{1}=1\)).

Figure 10: _Resnet20_ and _Resnet8_ residual blocks

Figure 9: Buffer partitioning, \(\textit{ow}_{i}^{\text{par}}=2\). The data in output from each _task Ti_ is connected as input to the FIFO slice at the position \(i+\textit{2}\) because of activation reuse.
from the previous stage until the first output activation is generated by the convolution and the merged output can be generated. In previous dataflow implementations of CNNs, this buffering consumed large amounts of memory ( [34]).

To efficiently support _residual_ blocks, the multiple endpoints of the input tensor and the increased buffering caused by the different number of convolutions (and thus different computation delays) per branch must be handled differently.

The combination of the following optimization steps for the dataflow architecture, _proposed for the first time in this paper_, can avoid it, e.g., in CNNs such as Resnet8 and Resnet20.

The following two transformations (see Fig. 12) show how to solve the problem of multiple endpoints, reducing the length of the skip connection and the required buffering:

* Loop merge: if the residual block has a downsample layer, i.e., a pointwise convolution in the short branch of the skip connection, the algorithm merges the two convolution loops. Both computations are performed by the same task, which provides the tensor produced by the merged layer as an additional output.
* Temporal reuse: to avoid buffering the same tensor twice, if the residual block does not have a downsample layer, the graph optimization uses the window buffer as input to the convolution. The values are forwarded, using a second output stream, to the next layer once they have been completely used.

Thanks to these transformations, the two input streams of the _add_ merge layer are produced simultaneously, and computation tasks never stall. _Conv0_ writes the skip connection stream as soon as the convolution computation starts and at the same rate as the convolution output tensor. The last transformation, shown in Fig. 13, removes the sum of the value coming from the short branch by connecting it as an additional contribution to the second convolution of the long branch. The value from the skip branch is used to initialize the accumulator register, and the addition operation is removed from the network graph.

The producer and consumer of the two branch streams are now the same (_conv0_ and _conv1_), and they produce/consume at the same rate. The required buffering of the skip connection (\(B_{\mathrm{sc}}\)) is now equal to the _conv1_ window buffer size

\[B_{\mathrm{sc}}=B_{1}=[(\textit{fh}_{1}-1)\cdot\textit{iw}_{1}+\textit{fw}_{ 1}-1]\cdot\textit{ich}_{1}. \tag{22}\]

The dimensions of the first residual block without downsample of _Resnet20_ are: \(\textit{iw}_{0}=\textit{iw}_{1}=32\), \(\textit{ich}_{0}=\textit{ich}_{1}=16\), \(\textit{fh}_{0}=\textit{fh}_{1}=3\), \(\textit{fw}_{0}=\textit{fw}_{1}=3\).

The dimensions of the first residual block with downsample of _Resnet20_ are: \(\textit{iw}_{0}=32\), \(\textit{iw}_{1}=16\), \(\textit{ich}_{0}=16\), \(\textit{ich}_{1}=32\), \(\textit{fh}_{0}=\textit{fh}_{1}=3\), \(\textit{fw}_{0}=\textit{fw}_{1}=3\).

The skip connection buffering, \(B_{\mathrm{sc}}\), is then reduced to \(R_{\mathrm{sc}}\), in both cases

\[R_{\mathrm{sc}}=\frac{[(\textit{fh}_{1}-1)\,\textit{iw}_{1}+\textit{fw}_{1}- 1]\,\textit{ich}_{1}}{[(\textit{rh}_{0}-1)\,\textit{iw}_{0}+\textit{rw}_{0}] \,\textit{ich}_{0}}=0.5. \tag{23}\]

The same calculated gain holds for all residual blocks in _Resnet20_ because the product \(\textit{iw}_{i}\cdot\textit{ich}_{i}\) remains constant. The same considerations apply to _Resnet8_, since the structure of its residual blocks is identical to those already analyzed.

From a network graph of a residual block with and without downsampling, Fig. 14 shows the initial and final representations after applying the previously described optimizations.

## 4 Experimental Results

Our architecture is evaluated on the CIFAR-10 dataset, which consists of \(32\)x\(32\) RGB images. The same preprocessing and data augmentation used in [15] is used for both training and testing. The model is trained for \(400\) epochs with a batch size of \(256\), using the stochastic gradient descent (SGD) optimizer and cosine annealing as the learning rate scheduler.

Synthesizable C++ is used for both the hand-written layer process library and the Python-generated top-level dataflow code that calls the process functions. The implementation

Figure 12: Multiple endpoint graph optimizations: (a) input forwarding without downsampling, (b) layer merging when there is a downsample pointwise convolution.

Figure 13: The addition is optimized as initialization of the convolution accumulator register.
flow uses Xilinx Vitis HLS for RTL code generation and Vivado for implementation on the Ultra96-v2 and Kria KV260 boards. Table 2 shows the available resources for the two boards.

The obtained throughputs (FPS, \(\mathrm{Gops/s}\)) and the latency (\(\mathrm{ms}\)) are shown in Table 3. The final resource utilization is summarized in Table 4.

Our proposed architecture is first compared with a ResNet20 implementation and the derived AdderNet described in [42], which is are the most efficient CNN implementations on FPGAs in terms DSP packing and model architecture to-date. Our implementation achieves speedups (\(\mathrm{Gops/s}\)) of \(2.88\)x and \(1.94\)x with \(0.5\,\%\) and \(1.4\,\%\) higher accuracy, with respect to the ResNet20 and Addernet in [42], using the Kria KV260 as a reference platform. Also, the latency is reduced by \(3.84\)x and \(1.96\)x respectively.

We then compare our results with the implementations of the ResNet8 model by Vitis AI and FINN described in [14]. Our solution achieves speedups of \(6.8\)x and \(2.2\)x with a latency improvement of \(28.1\)x and \(3.35\)x  respectively. Vitis AI achieved better accuracy by \(0.5\,\%\), probably because it executes _batch normalization_ in hardware, while our implementation outperformed a 4-bit FINN implementation by \(2.8\,\%\).

## 5 Conclusion

This work presents a design flow for CNNs specifically optimized for residual networks. It supports the most commonly used operations for classic CNNs, including convolutions, fully connected (linear) layers, batch normalization, ReLU activation functions, max/average pooling, and skip connections. It is also fairly platform-independent, since it is based on heavily templatized layer models and comes with an ILP-based optimization method to maximize throughput under resource constraints. This allows it to be used with various FPGA platforms, including embedded ones.

A dataflow pipelined architecture minimizes buffering resources for networks with skip connections. The design is validated by experiments on ResNet8 and ResNet20 using the CIFAR-10 dataset. Both activations and weights are quantized in INT8 format using power-of-two scaling factors.

The design uses PyTorch and Brevitas for training and quantization, and Vitis HLS and Vivado for hardware implementation on Kria KV-260 and Ultra96-v2 boards.

The solution achieves an accuracy of \(88.7\,\%\) for ResNet8 and \(91.3\,\%\) for ResNet20, with throughputs of \(12\,971\) FPS and \(3254\) FPS on the Ultra96, and \(30\,153\) FPS and \(7601\) FPS on the Kria KV260. Compared to the state-of-the-art for CNN residual network acceleration on FPGAs [42], it achieves \(2.88\)x speedup with \(0.5\,\%\) higher accuracy for ResNet20 and \(2.2\)x speedup with \(2.8\,\%\) higher accuracy for ResNet8 [14]. Compared to a residual network with packed adders [42], it achieves \(1.94\)x speedup with \(1.4\,\%\) higher accuracy and a latency improvement of \(1.96\,\times\). Considering state-of-the-art frameworks, the comparison shows that the resource-efficient implementation of the residual layer achieves a Pareto-optimal implementation for accuracy, throughput, and latency. Since the boards are the same and all approaches utilize most resources of each FPGA, lower latency also means lower energy than the state of the art.

In summary, the proposed design architecture shows potential as an alternative to the commonly used residual network accelerators on platforms with limited resources. It delivers greater throughput and energy efficiency than the state-of-the-art without increasing hardware costs.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Board** & **FPGA part** & **LUT** & **FF** & **BRAM** & **DSP** & **URAM** \\ \hline Ultra96 & xcza3eg & 141120 & 70560 & 216 & 360 & 0 \\ Kria KV260 & xcza5eg & 234240 & 117120 & 144 & 1248 & 64 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Resources of the Ultra96 and Kria KV260 boards

Figure 14: Graph optimization for the residual blocks of Resnet8 and Resnet20 networks. The skip connection goes through the first convolution layer, conv0, into the second convolution layer, conv1, reducing buffering requirements with and without downsampling.