EfficientRep:An Efficient Repvgg-style ConvNets with Hardware-aware
  Neural Network Design
We present a hardware-efficient architecture of convolutional neural network,
which has a repvgg-like architecture. Flops or parameters are traditional
metrics to evaluate the efficiency of networks which are not sensitive to
hardware including computing ability and memory bandwidth. Thus, how to design
a neural network to efficiently use the computing ability and memory bandwidth
of hardware is a critical problem. This paper proposes a method how to design
hardware-aware neural network. Based on this method, we designed EfficientRep
series convolutional networks, which are high-computation hardware(e.g. GPU)
friendly and applied in YOLOv6 object detection framework. YOLOv6 has published
YOLOv6N/YOLOv6S/YOLOv6M/YOLOv6L models in v1 and v2 versions.
Kaiheng Weng, Xiangxiang Chu, Xiaoming Xu, Junshi Huang, Xiaoming Wei
# EfficientRep: An Efficient Repvgg-style ConvNets with Hardware-aware Neural Network Design

###### Abstract

We present a hardware-efficient architecture of convolutional neural network, which has a repvgg-like architecture. Flops or parameters are traditional metrics to evaluate the efficiency of networks which are not sensitive to hardware including computing ability and memory bandwidth. Thus, how to design a neural network to efficiently use the computing ability and memory bandwidth of hardware is a critical problem. This paper proposes a method how to design hardware-aware neural network. Based on this method, we designed EfficientRep series convolutional networks, which are high-computation hardware(e.g. GPU) friendly and applied in YOLOv6 object detection framework. YOLOv6 has published YOLOv6N/YOLOv6S/YOLOv6M/YOLOv6L models in v1 and v2 versions. Our YOLOv6 code is made available at [https://github.com/meituan/YOLOv6](https://github.com/meituan/YOLOv6).

 neural-network-design, hardware-aware

## I Introduction

Since VGG achieved success in image classification tasks, convolutional neural network design has attracted huge attentions in academy and industry. Currently, amounts of classical networks have been proposed, like Inception [1] and Resnet [2]. These well-designed architectures make the accuracy of image classification increasingly higher. Besides manually design, recently neural architecture search also automatically designed several representative networks, such as Nasnet [3] and AmoebaNet [4]. Though complexed networks bring successes to vision tasks like image classification, object detection and segmentation, these networks may not get suitable accuracy-speed balance on deployed hardware.

Deep learning network design and deployment for hardware efficiency has been consistently studied [5][6]. Traditional evaluation metrics for inference effciency are floating-point operations (FLOPs) and parameter count. However, such metrics can not represent the relationships with hardware, such as memory access cost and I/O throughput. Fig 3 shows the relationships between computing ability with memory resources. Hence, an important question is raised to us: _how to design a hardware-friendly network to perform better accuracy-speed balance?_

To solve this problem, we explored some novel architectures that show competitive and applied in YOLOv6 object detection framework. RepVGG [7] is a novel network with 3x3 convolutional kernel highly optimized by winograd algorithm on GPU or CPU. Single-path models can train and infer fast on devices like GPU. Fig 4. shows the transformation between training state and infenrece state for rep conv. In training state, with extra 1x1 conv and identity, rep conv can guarantee the accuracy during training. In inference state, re-parameterization structure can be equally converted to inference status.

In YOLOv6-1.0, based on basic rep conv, we designed pure regvgg-style architectures named EfficientRep backbone and Rep-PAN neck to efficiently utilize the computing resources of GPU. In YOLOv6-2.0, to balance the computing and memory resources, we explored a kind of novel structure named Rep(Beer-mug) unit and RepC3(or named CSPStackRep) block. Compared with rep conv, we found that Rep unit is a more efficient basic neural network unit at some states. Though RepVGG proposed that rep-style multi-branch training can reach comparable performance as an original multi-branch training like resnet, we discovered that the accuracy-speed trade-off of rep-style network degraded at large scale which will be shown in Section III. Hence, we designed CSPRep backbone and CSPRepPAN neck applied in YOLOv6-2.0, with above structures of Rep unit and RepC3 block.

With above considerations, we apply hybrid strategy to YOLOv6 that we select single-branch models in small size and multi-branch models in large size. Fig 1 and Fig 2 describe the comparable accuracy-speed trade-off as other object detectors.

Fig. 1: Metrics of YOLOv6-v1.

Fig. 2: Metrics of YOLOv6-v2.
## II Related Work

### _Neural Network Design_

VGG [8] network achieved the top-1 accuracy of ImageNet Classification to above 70%, many related innovations have been proposed, e.g. GoogLeNet [9] and Inception [1] network are designed in multi-branch architecture. ResNet [2] is a representative two-branch network and widely used in industry. Until repvgg [7] is proposed, single-path network shows high efficiency on some devices.

### _Neural Architecture Search_

Neural architecture search (NAS) is a novel technique, aiming to automatically design network compared with manual designing. With manual-designing space design, NAS can automatically generate numbers of networks with huge resource cost. Currently, to save computing resources, low-cost neural network search has been proposed, e.g. One-For-All [10] et al.

### _Hardware-aware Neural Network Design_

Recently, serveral novel networks like MobileOne [5] and TRT-VIT [6] have been proposed. These networks are manually designed, considering the performance on devices. Except for the metrics about accuracy or params, the inference speed is taken into accounts simultaneously, which is called hardware-aware neural network design.

#### Our contributions are summarized as follows.

* We proposed novel structures of Rep unit, Reblock and RepC3 block.
* We proposed novel network of EfficientRep, Rep-PAN, CspBep and CSPRepPAN.
* We proposed novel network design with computing ability and memory bandwidth balance, with various strategies for different size of models.

## III Approach

In this section, we describe the details of hardware-aware neural network design applied in YOLOv6. The novel structures and networks we proposed will be presented with hybrid strategy for different size of models.

### _Pure Repvgg-style Efficient Design_

Repvgg-style [7] conv has the structure of 3x3 conv followed by ReLU, which can efficiently utilize the hardware computation. At training state, Repvgg-style conv consists of 3x3 branch, 1x1 branch and identity(Fig 4). Through re-parameterization, multi-branch structure is transformed to single-branch 3x3 conv at inference state. As shown in Fig 5 and Fig 6, we designed Repvgg-style network called EfficientRep backbone and Rep-PAN neck which are GPU-friendly, and applied in YOLOv6 detection framework(YOLOv6-v1) [11].

However, when YOLOv6-v1 grows to medium size, the inference speed slows down too fast and the accuracy is not competitive, compared with csp-style YOLO-series [12][13][14][15][16][17][18]. As shown in Table I, pure repvgg-style YOLOv6m can not achieve comparable accuracy-speed trade-off. Hence, we explored novel structure like multi-path for large size models.

Fig. 4: Design of Rep Conv.

Fig. 5: Design of EfficientRep.

Fig. 3: Introduction of Roofline Model.
### _Multi-path Efficient Design_

To solve the problem that pure repvgg-style network can not get accuracy-speed trade-off as expected, we proposed a novel structure of Bep unit. The details of Bep unit is shown in Fig 7, numbers of rep convs are connected linearly with extra shortcut. With Bep unit, we designed a novel backbone and neck, separately named CSPBep and CSPRepPAN. We applied the above structures in YOLOv6-v2 and achieved better accuracy-speed trade-off.

CSP-style [20] is an efficient design widely used in YOLOverse framework, like YOLOv5 [16], PPYOLOE [17] and so on. CSP-style structure uses Cross Stage Partial Network which achieves a richer gradient combination while reducing the amount of computation. We combined Bep unit with CSP-style structure to design a novel structure named BepC3 block to balance accuracy and inference speed. The design of BepC3 is described in Fig 8, which is composed of CSP-style structure and Rephlock of Bep units. As Table I shows the improvements of BepC3 block, the accuracy and speed is balanced as expected.

Based on BepC3 block, we separately designed CSPRep backbone and CSPRepPAN neck which result in the YOLOv6-v2 models. For CSP-style network in YOLO-series, the partial ratio is 1/2 by default. In our design for YOLOv6-v2, we applied partial ratio of 2/3 for YOLOv6m and 1/2 for YOLOv6l, aiming to get better performance.

### _Scaling Strategy_

Following with YOLOv5, we use scale strategy of depth multiplier and width multiplier to generate various size of models. In YOLOv6-v1 and YOLOv6-v2, the depth settings of basic backbones are both [1, 6, 12, 18, 6]. Besides, the width settings are [64, 128, 256, 512, 1024]. The depth setting of basic necks are both [12, 12, 12, 12] and the width settings are [256, 128, 128, 256, 512, 512]. Table II shows the specific depth multiplier and width multiplier applied in YOLOv6.

## IV Experimental Results

In this section, we show experimental results and details. For object detection, experiments are trained on MS COCO-2017 training set with 80 classes and 118k images. We use standard COCO AP metric on MS COCO-2017 validation set with 5000 images.

### _Experiments Details for Object Detection_

For our network design applied in YOLOv6, we train the models for a total of 300 epochs with 3 epochs warmup on

Fig. 8: Design of BepC3.

Fig. 6: Design of Rep-PAN.

Fig. 7: Design of Bep Unit.
COCO train2017. Our training strategy is stochastic gradient descent (SGD) for training and an initial lr of 0.01 with the cosine lr scheduler. In our setting, the weight decay is 5e-4. For 8-GPU device, the batch size is 256 by default. We also adopt Mosaic and Mixup data augmentations and exponential moving average (EMA) during training.

### _Comparision with Other Detectors_

To present the effects of our network design, Table III shows the performance of YOLOv6 models, compared with other state-of-the-art object detectors on MS-COCO test split. With our optimized models and other improvements, YOLOv6-N/S/M/L models present better accuracy-speed trade-off. We evaluate inference speed on NVIDIA Tesla T4 GPU with TensorRT version 7.2, with FP16 precision.

## V Conclusion

In this report, we present our optimization for neural networks applied in YOLOv6. We designed Bep unit, RepC3-block/Repl block, EfficientRep/CspBep backbone and RepPAN/CSPRepPAN neck. Joined with other improvements, we achieved YOLOv6-N/S/M/L models better than other object detectors. Meanwhile, we proposed a hybrid network-design strategy applied in various size of models, aiming to achieve better accuracy-speed trade-off. Moreover, we proposed a novel hardware neural network design with computing and memory balance, which is applied in YOLOv6 framework development. We hope that the above proposals can provide inspirations for develops and researchers.
