Data-Free Backbone Fine-Tuning for Pruned Neural Networks
Model compression techniques reduce the computational load and memory
consumption of deep neural networks. After the compression operation, e.g.
parameter pruning, the model is normally fine-tuned on the original training
dataset to recover from the performance drop caused by compression. However,
the training data is not always available due to privacy issues or other
factors. In this work, we present a data-free fine-tuning approach for pruning
the backbone of deep neural networks. In particular, the pruned network
backbone is trained with synthetically generated images, and our proposed
intermediate supervision to mimic the unpruned backbone's output feature map.
Afterwards, the pruned backbone can be combined with the original network head
to make predictions. We generate synthetic images by back-propagating gradients
to noise images while relying on L1-pruning for the backbone pruning. In our
experiments, we show that our approach is task-independent due to pruning only
the backbone. By evaluating our approach on 2D human pose estimation, object
detection, and image classification, we demonstrate promising performance
compared to the unpruned model. Our code is available at
https://github.com/holzbock/dfbf.
Adrian Holzbock, Achyut Hegde, Klaus Dietmayer, Vasileios Belagiannis
# Data-Free Backbone Fine-Tuning for Pruned Neural Networks

###### Abstract

Model compression techniques reduce the computational load and memory consumption of deep neural networks. After the compression operation, e.g. parameter pruning, the model is normally fine-tuned on the original training dataset to recover from the performance drop caused by compression. However, the training data is not always available due to privacy issues or other factors. In this work, we present a data-free fine-tuning approach for pruning the backbone of deep neural networks. In particular, the pruned network backbone is trained with synthetically generated images, and our proposed intermediate supervision to mimic the unpruned backbone's output feature map. Afterwards, the pruned backbone can be combined with the original network head to make predictions. We generate synthetic images by back-propagating gradients to noise images while relying on L1-pruning for the backbone pruning. In our experiments, we show that our approach is task-independent due to pruning only the backbone. By evaluating our approach on 2D human pose estimation, object detection, and image classification, we demonstrate promising performance compared to the unpruned model. Our code is available at [https://github.com/holzbock/dfbf](https://github.com/holzbock/dfbf).

 Data-Free Neural Network Pruning, Data-Free Object Detection Pruning, Data-Free Pose Estimation Pruning

## I Introduction

Computer vision tasks like object detection [1, 2] or human pose estimation [3, 4] have long been studied in the literature. Their leading performance though comes at the cost of high computational and memory requirements, making them difficult to deploy on resource-constrained devices. For deep neural networks, the compute and memory demands are often reduced with model compression, e.g. parameter pruning [5] or knowledge distillation [6]. However, most neural network compression techniques normally require the availability of the training set. Due to privacy issues or large dataset size, access to the training dataset cannot always be guaranteed.

As an alternative to the data-driven model compression [5, 6, 7], approaches using only a few samples of the training dataset [8] and data-free methods have been developed to overcome the limited access to the original training dataset. The data-free approaches generate synthetic images to regain the knowledge after pruning [9] or transfer the knowledge with knowledge distillation [10] to a smaller model. The synthetic images can be generated with an additional generator network trained with the original model's knowledge [9, 11]. Another method is to optimize noisy images by back-propagating directly onto the pixels [12]. In this work, we focus on neural network pruning similar to [9]. However, we rely on the back-propagation of gradients for image generation instead of employing a generator network. In contrast to existing methods, our approach is not limited to a specific task, but is applicable to different computer vision tasks. We achieve task-independence by only pruning the backbone of the model, keeping the task-dependent head unchanged.

We present a **D**ata-**F**ee **B**ackbone **F**ine-tuning approach (DFBF) for pruning the backbone of deep neural networks. In particular, we train the pruned backbone with synthetic images to mimic the output of the unpruned backbone. Later, we can use the pruned backbone with the original network head to make predictions. The synthetic images are generated

Fig. 1: Training of the pruned backbone with label-free synthetic images. Besides the backbone’s output feature map, some intermediate feature maps are used to recover the backbone’s knowledge.
by back-propagating gradients directly on noise images, while the backbone is pruned by \(\ell_{1}\)-pruning, leaving the parameters of the network head unmodified. Only backbone pruning keeps the pruning approach task-independent, as no training of the task-specific head is required. For the training of the backbone, we introduce an intermediate loss function that adjusts the pruned backbone's intermediate feature maps and output to match it with the original backbone. An overview of the proposed method is given in Fig. 1. After the training, the original network's head can be attached to the pruned backbone for the task-specific output.

We examine our approach on three vision tasks, namely object detection, human pose estimation, and image classification. First, we use the Faster R-CNN object detector [1] for object detection and present the results for different pruning ratios on the COCO detection dataset [13] and Pascal VOC dataset [14]. We evaluate pose estimation with the OpenPifPaf pose estimator [3] trained on the COCO keypoint dataset [13]. Additionally, we compare the performance on CIFAR10 [15] for image classification. As backbone, we use ResNet [16] and VGG [17] models in our experiments. To the best of our knowledge, we are the first to introduce a data-free fine-tuning method for pruning that can be applied to complex computer vision tasks like object detection and human pose estimation. The proposed intermediate loss function that applies the prediction of the unpruned model as ground truth enables the task independence of DFBF.

## II Method

In the following, we present our data-free fine-tuning method for pruned neural networks, which process input images \(\mathbf{x}\in\mathbb{R}^{3\times W\times H}\), where \(W\) and \(H\) are the width and height, respectively. We assume that the neural network \(\mathbf{y}=f(\mathbf{x};\theta)\) can be divided into a backbone \(\mathbf{z}=b(\mathbf{x};\theta_{b})\) and a head \(\mathbf{y}=h(\mathbf{z};\theta_{h})\), where the head \(h\) is a task-specific output network. \(\mathbf{y}\) is the output prediction, \(\mathbf{z}\) is the output feature map from the backbone forwarded to the task-specific head, while \(\theta\), \(\theta_{b}\), and \(\theta_{h}\) are the model parameters of the whole model, the backbone, and the head, respectively. Since we only prune the model backbone, our approach is independent of the shape of the model prediction \(\mathbf{y}\) and thus can be applied to different computer vision tasks.

### _Preliminaries_

PruningUtilizing pruning algorithms reduces neural networks' resource demand by removing unnecessary or redundant parameters. Pruning convolutional neural networks can be divided into different groups. The parameter pruning only removes single weights [18], leading to sparse kernels and no satisfying reduction in computation. In contrast, filter pruning reduces the computation effort by removing entire filters [19]. To decide which filters are pruned, the methods rely on the knowledge of the model, like in \(\ell_{1}\)-pruning [20] and Batch Normalization pruning [21], or even use additional neural network layers [22]. Because of its simplicity and the fact that no extra neural network layers are needed, we rely on the \(\ell_{1}\)-pruning [20] to get the pruned backbone \(b_{p}(\cdot)\) with its reduced parameters \(\theta_{bp}\). However, every other filter pruning approach can be used to prune the model's backbone for DFBF. The \(\ell_{1}\)-pruning calculates for each filter \(\mathcal{F}_{i}\) of a neural network layer the sum of the filter weights \(w_{i}\) by \(s_{i}=\sum|w_{i}|\) and removes filters with the smallest sum \(s_{i}\) according to the required sparsity. The reduction of the current layer's filters makes it necessary to remove the corresponding input channels of the following layer. Furthermore, the pruning sparsity can be adapted to the number of filters in the layer, i.e. in layers with many filters, percentual more filters are pruned than in layers with fewer filters.

Image SynthesisSince the original training set is unavailable, we synthesize training images by transferring knowledge from the model to a noisy image, similar to DeepInversion [23], but without being limited to the classification task. At the beginning of the image generation, a noise image \(\mathbf{\hat{x}}\in\mathcal{R}^{3\times w\times h}\) with width \(w\) and height \(h\) is fed into the backbone, while the synthetic image size can differ from the original \((W\neq w;H\neq h)\). Then, the loss \(\mathcal{L}_{image}\) is propagated back onto the noise image \(\mathbf{\hat{x}}\) to adapt the pixels directly without changing the model parameters. The loss \(\mathcal{L}_{image}\) for optimizing the noise image independent of the underlying task is the weighted sum of the following three parts. The Batch Normalization loss is calculated between the batch statistics of the Batch Normalization layers and the statistics of the actual noise images. The total variance loss is defined by the \(\ell_{2}\)-norm of the differences between horizontally and vertically adjacent image pixels and is calculated on the noise image directly. Additionally, the loss is regularized by penalizing the \(\ell_{2}\)-norm of the entire image. The task independence of \(\mathcal{L}_{image}\) is reached by not using the task-specific outputs of the head, whereby images of different computer vision tasks can be synthesized. To train the pruned model, we generate a synthetic dataset \(\mathcal{D}\) containing \(M\) synthetic images \(\mathbf{\hat{x}}\).

### _Overall Pruning Procedure_

Our method is designed for pruning a neural network of any image-dependent task in a data-free manner. Our main contribution is the fine-tuning step after the pruning without relying on training data. Therefore, we focus on reducing the size of the backbone \(b(\cdot)\) and keeping the head \(h(\cdot)\) as is. We perform the task-independent data-free pruning in the following three steps: 1) image generation with an adapted loss function, 2) network pruning with the \(\ell_{1}\)-pruning, and 3) fine-tuning only the backbone with the proposed intermediate loss function. A systematic overview of the data-free fine-tuning of the backbone is given in Fig.1.

### _Data-Free Training_

During the pruning process, the model loses some of its knowledge which can be recovered in the following training procedure. In data-driven pruning methods, fine-tuning is performed with the original training dataset. In contrast, we use the synthetic dataset \(\mathcal{D}\), which only contains pseudo images \(\mathbf{\hat{x}}\), but no labels. However, the loss calculation in the standard
training with the original data needs labels for the parameter update. To overcome the issue of having no ground truth labels for the pseudo images, we introduce a method that generates pseudo labels with the help of the unpruned model and applies them to improve the performance of the pruned model.

The final output of a neural network depends on the defined model task. Therefore, we propose not handling the output of the unpruned network's head as pseudo ground truth but instead using the output feature map \(\mathbf{z}\) of the unpruned model's backbone. This design choice makes our approach independent of the task-specific model heads. More precisely, we define the loss as the \(\ell_{1}\)-loss between the output feature map of the unpruned backbone \(\mathbf{z}\) and the pruned backbone \(\mathbf{\hat{z}}\). Our output feature map loss \(\mathcal{L}_{out}\) can be formulated as

\[\mathcal{L}_{out}=\frac{1}{w_{o}*h_{o}}\sum_{i=0}^{w_{o}}\sum_{j=0}^{h_{o}}| \mathbf{\hat{z}}_{i,j}-\mathbf{z}_{i,j}|, \tag{1}\]

where \(w_{o}\) and \(h_{o}\) are the width and height of the output feature map, respectively.

Additionally, we define an intermediate feature map loss \(\mathcal{L}_{inter}\). For \(\mathcal{L}_{inter}\), we calculate the \(\ell_{1}\)-loss between intermediate feature maps of the unpruned model \(\mathbf{a}_{n},n\in[1\dots N]\) and the corresponding feature maps of the pruned model \(\mathbf{\hat{a}}_{n},n\in[1\dots N]\), where \(N\) defines the number of intermediate feature maps. Empirically, we found that the intermediate feature maps after the Batch Normalization layers [24] lead to the best results. Since the feature maps behind the pruned layers differ in their dimensions from the unpruned model ones, it is impossible to calculate the loss between them. Therefore, we propose to skip single layers in the pruning and use them later for the loss calculation. Using \(N\) intermediate feature maps for the loss calculation, \(\mathcal{L}_{inter}\) can be calculated as

\[\mathcal{L}_{inter}=\sum_{n=1}^{N}\left(\frac{\mu_{n}}{w_{n}*h_{n}}\sum_{i=0}^{ w_{n}}\sum_{j=0}^{h_{n}}|\mathbf{\hat{a}}_{n,i,j}-\mathbf{a}_{n,i,j}|\right). \tag{2}\]

The influence of the different feature maps on the loss \(\mathcal{L}_{inter}\) is defined by the parameter \(\mu_{n}\), which we define as \(\frac{n}{N+1}*\gamma+1\), where the parameter \(\gamma\) is a scaling factor. \(w_{n}\) and \(h_{n}\) are the width and height of the intermediate feature map \(n\).

The overall loss \(\mathcal{L}_{DFBF}\) in our data-free training combines \(\mathcal{L}_{out}\) and \(\mathcal{L}_{inter}\):

\[\mathcal{L}_{DFBF}=\mu_{out}\mathcal{L}_{out}+\mathcal{L}_{inter}, \tag{3}\]

where \(\mu_{out}\) is the weighting factor of the output loss that we define as \(\gamma+1\). Importantly, \(\mathcal{L}_{DFBF}\) is applied to optimize the parameters \(\theta_{bp}\) of the pruned backbone \(b_{p}(\cdot)\) and not to update the parameters \(\theta_{h}\) of the head \(h(\cdot)\).

During inference, we combine the optimized pruned backbone \(\mathbf{\hat{z}}=b_{p}(\mathbf{x},\theta_{bp})\) with the pre-trained head \(\mathbf{\hat{y}}=h(\mathbf{\hat{z}},\theta_{h})\) to obtain the pruned model as:

\[\mathbf{\hat{y}}=h(b_{p}(\mathbf{x},\theta_{bp}),\theta_{h}). \tag{4}\]

For the prediction, we feed original images \(\mathbf{x}\) to the model and get a slightly different model output \(\mathbf{\hat{y}}\) because of the modified backbone weights \(\theta_{p}\). Due to the backbone training with the synthetic images, the output of the original model \(\mathbf{y}\) and the pruned model \(\mathbf{\hat{y}}\) are nearly identical.

## III Experiments

We show the effectiveness of our proposed method in three challenging computer vision tasks: object detection, human pose estimation, and image classification. In object detection, we use Faster R-CNN [1] trained on the COCO detection [13] as well as on the Pascal VOC dataset [14]. The base for the human pose estimation is an Open PifPaf pose estimator [3] trained on the COCO keypoint dataset [13]. Both tasks utilize a ResNet50 [16] backbone, and VGG16 [17] is used as an additional backbone in object detection. We prune between 10% and 40% of the backbone filters with \(\ell_{1}\)-pruning during the evaluation. In image classification, we compare with Tang et al. [9] and follow their evaluation protocol using Batch Normalization pruning [21] for the VGG models [17] and \(\ell_{1}\) pruning [20] for the ResNet models [16]. The pruned backbone is trained in each task with 1600 synthetic images using the SGD optimizer with a learning rate of 0.01, a momentum of 0.9, and a weight decay of \(5\times 10^{-4}\).

To the best of our knowledge, we are the first to prune an object detection or a human pose estimation model in a data-free manner. Therefore, we cannot make a direct comparison with other methods. We compare DFBF with the unpruned baseline referred to as _Baseline_, the pruned but not fine-tuned model referred to as _w/o fine-tuning_, and the pruned model trained with 1600 random sampled original training images and our intermediate loss \(\mathcal{L}_{DFBF}\) referred to as _orig. img._ Additionally, we report the number of removed filters and parameters, while the parameter count differs from the filter count because of different pruning spars

Fig. 3: Synthetic images generated with an OpenPifPaf pose estimator [3] with a ResNet50 [16] backbone trained on the COCO keypoint dataset [13].

Fig. 2: Synthetic images generated with Faster R-CNN object detector [1] with a ResNet50 [16] backbone trained on the COCO detection dataset [13].
### _Object Detection_

We set the synthetic image resolution for the object detection task to \(250\times 250\) pixels and \(\gamma\) to 1. The results for pruning a Faster R-CNN object detection model [1] in a data-free manner are shown in Tab. I. For both datasets and backbones, the performance evolves similarly during pruning and training. The performance of the object detector after pruning depends on the pruning sparsity and decreases with higher rates. Also, we can see that with DFBF, the performance of the pruned model can recover during training with the synthetic images near the initial results. The model's performance trained with synthetic images is behind the model trained with the original images. Furthermore, the influence of synthetic images at higher pruning rates (30% and 40%) can be seen. Here, the difference between the model trained with the synthetic and the original images increases compared to lower pruning rates (10% and 20%). We show some synthetic images generated from an object detector trained on the COCO detection dataset [13] in Fig. 2. Compared to the original images, no objects can be recognized in the synthetic images. The abstract look of the synthetic images can cause the performance gap between the synthetic and original training image's performance using high pruning sparsities.

### _Human Pose Estimation_

Besides object detection, we evaluate DFBF for human pose estimation which is often the base for gesture recognition [25] or human motion prediction [26]. We use synthetic images with a resolution of \(160\times 160\) and 6 as value for \(\gamma\). The results for the human pose estimation are shown in Tab. I. As for object detection, the performance decreases with a higher pruning rate in human pose estimation. For all pruning rates, DFBF can recover the performance after pruning. In contrast to object detection, we perform better on human pose estimation with the synthetic images than with the original training images. This could be due to the difference between the synthetic images obtained from both tasks. Synthetic images generated with OpenPifPaf trained on the COCO keypoint dataset can be seen in Fig. 3. Compared to the images generated in the object detection task in Fig. 2, the pose images show pose information and pose-related details such as faces. Therefore, the generated images' variation can be broader compared to the original images. Moreover, the synthetic images show only patterns important for human pose estimation, and the pruned model can concentrate on learning the essential shapes.

### _Image Classification_

Unlike the other tasks, we compare image classification with the state-of-the-art approach from Tang et al. [9]. During pruning the image classification model, the resolution of the synthetic training images is \(32\times 32\), and we set the factor \(\gamma\) to 0. Tab. II presents the CIFAR10 [15] results for different ResNet and VGG models fine-tuned with DFBF and the approach of Tang et al. [9]. In the pruning, we remove 30% and 50% of the filters from the baseline models. The results show that our approach is on par with Tang et al. [9], while we are not limited to the classification task due to the proposed task-agnostic loss function.

## IV Ablation Studies

An essential part of DFBF is the proposed intermediate loss function \(\mathcal{L}_{DFBF}\) for which we do further investigations. We use the OpenPifPaf settings from Sec. III as standard settings.
In \(\mathcal{L}_{DFBF}\), we take the feature map after each residual connection and the output feature map to calculate the loss and get an overall performance of 62.2%. The performance drops by 1.0% AP to 61.2% AP when skipping every second intermediate feature map in the loss calculation. Neglecting all intermediate feature maps and using only the output feature map in the loss calculation decreases the performance to 44.7% AP. These experiments demonstrate the importance of the intermediate feature maps on \(\mathcal{L}_{DFBF}\). Furthermore, we present the effect of the scaling factor \(\gamma\) in Tab. III. We vary the impact of the different feature maps on \(\mathcal{L}_{DFBF}\) by setting \(\gamma\in[2,\dots,8]\). With increasing \(\gamma\), the overall performance improves.

## V Conclusion

We presented a data-free backbone fine-tuning approach for pruning the backbone of deep neural networks. Our approach relies on synthetically generated images to fine-tune the backbone of the pruned neural network, where pruning is based on the \(\ell_{1}\) norm. Notably, we proposed an intermediate loss function to match the pruned backbone's output feature map such that the pruned backbone can later be combined with the original network head to perform predictions. Our evaluations showed that our approach is task-independent by evaluating the tasks of object detection, human pose estimation, and image classification.
