Domain-Aware Fine-Tuning: Enhancing Neural Network Adaptability
Fine-tuning pre-trained neural network models has become a widely adopted
approach across various domains. However, it can lead to the distortion of
pre-trained feature extractors that already possess strong generalization
capabilities. Mitigating feature distortion during adaptation to new target
domains is crucial. Recent studies have shown promising results in handling
feature distortion by aligning the head layer on in-distribution datasets
before performing fine-tuning. Nonetheless, a significant limitation arises
from the treatment of batch normalization layers during fine-tuning, leading to
suboptimal performance. In this paper, we propose Domain-Aware Fine-Tuning
(DAFT), a novel approach that incorporates batch normalization conversion and
the integration of linear probing and fine-tuning. Our batch normalization
conversion method effectively mitigates feature distortion by reducing
modifications to the neural network during fine-tuning. Additionally, we
introduce the integration of linear probing and fine-tuning to optimize the
head layer with gradual adaptation of the feature extractor. By leveraging
batch normalization layers and integrating linear probing and fine-tuning, our
DAFT significantly mitigates feature distortion and achieves improved model
performance on both in-distribution and out-of-distribution datasets. Extensive
experiments demonstrate that our method outperforms other baseline methods,
demonstrating its effectiveness in not only improving performance but also
mitigating feature distortion.
Seokhyeon Ha, Sunbeom Jung, Jungwoo Lee
# Domain-Aware Fine-Tuning: Enhancing Neural Network Adaptability

###### Abstract

Fine-tuning pre-trained neural network models has become a widely adopted approach across various domains. However, it can lead to the distortion of pre-trained feature extractors that already possess strong generalization capabilities. Mitigating feature distortion during adaptation to new target domains is crucial. Recent studies have shown promising results in handling feature distortion by aligning the head layer on in-distribution datasets before performing fine-tuning. Nonetheless, a significant limitation arises from the treatment of batch normalization layers during fine-tuning, leading to suboptimal performance. In this paper, we propose Domain-Aware Fine-Tuning (DAFT), a novel approach that incorporates batch normalization conversion and the integration of linear probing and fine-tuning. Our batch normalization conversion method effectively mitigates feature distortion by reducing modifications to the neural network during fine-tuning. Additionally, we introduce the integration of linear probing and fine-tuning to optimize the head layer with gradual adaptation of the feature extractor. By leveraging batch normalization layers and integrating linear probing and fine-tuning, our DAFT significantly mitigates feature distortion and achieves improved model performance on both in-distribution and out-of-distribution datasets. Extensive experiments demonstrate that our method outperforms other baseline methods, demonstrating its effectiveness in not only improving performance but also mitigating feature distortion.

## Introduction

Transferable neural network models have seen significant advancements in various domains, including computer vision [14, 15, 16], natural language processing [17, 18], and speech recognition [19, 20]. These models, trained on large-scale datasets with substantial computational resources, demonstrate exceptional performance and generalization capabilities, making them widely used in transfer learning. By employing these pre-trained models, transfer learning can substantially reduce the time and resources required for training and data collection, while also enhancing performance compared to training from scratch.

When transferring knowledge from a pre-trained model, the feature extractor is first initialized with the pre-trained model. Subsequently, there are two primary optimization methods commonly employed: linear probing (LP) and fine-tuning (FT). In LP, only the head layer is optimized, while the feature extractor is fixed during training. On the other hand, FT involves optimizing both the feature extractor and the linear head layer simultaneously. While FT generally outperforms LP on in-distribution (ID) datasets, recent findings show that FT performs worse on out-of-distribution (OOD) datasets compared to LP. It is due to feature distortion in pre-trained features that possess good generalization capabilities. To mitigate feature distortion, LP-FT was proposed as a two-stage training approach [17]. In the first stage, LP aligns the head layer with the ID subspace, and in the second stage, FT is performed with a small learning rate to mitigate feature distortion. This allows LP-FT to achieve improved performance on both ID and OOD datasets.

However, LP-FT still has several limitations. One of the key issues is the treatment of batch normalization layers during the FT stage in the LP-FT framework. During the FT stage, all batch normalization layers are switched to test mode, where fixed statistics are employed instead of batch statistics. While this prevents the alteration of batch normalization statistics during training, potentially mitigating feature distortion, it also harms beneficial effects such as improved gradient flow and reduced internal covariate shift [13]. Consequently, the overall optimization process is degraded, leading to suboptimal model performance. Furthermore, the initial LP stage in LP-FT optimizes the head layer with a feature extractor that has not been adapted to the target domain, which might not be ideal for domain adaptation. Moreover, employing a small learning rate during the FT stage restricts the optimization of the head layer. It becomes particularly severe when there is a significant dissimilarity in data distribution between the source and target domains. These limitations call for a more effective approach that can adequately address feature distortion while effectively performing batch normalization.

To address these challenges, we present a novel method named Domain-Aware Fine-Tuning (DAFT). Our method is built upon two core techniques: batch normalization conversion and the integration of LP and FT into a single stage. The batch normalization conversion method modifies batch normalization layers to better suit the target domain prior
to fine-tuning. It makes batch normalization more effective, reducing feature distortion by adjusting the statistics and parameters to the target domain. In addition, our DAFT integrates LP and FT into a single stage, where the linear head is optimized with gradually adapting features from the target domain. This strategy mitigates the issue of the head layer being optimized solely using a feature extractor that has not yet adapted to the target domain, resulting in improved optimization.

Through these techniques, our DAFT effectively reduces feature distortion and achieves enhanced performance compared to other baseline methods. The effectiveness of our method is demonstrated through similarity measures computed between features before and after adaptation. As shown in Figure 1, our DAFT consistently exhibits higher feature similarity compared to existing methods such as FT and LP-FT. Furthermore, we conduct extensive experiments to compare our proposed method against other baseline methods. These results demonstrate the superior performance of the proposed DAFT with higher accuracy and less feature distortion.

In summary, our main contributions are as follows.

* We propose a novel batch normalization conversion technique, which improves batch normalization during fine-tuning with reducing modifications to the neural network.
* The integration of LP and FT is proposed, allowing the head layer to be optimized with the gradual adaptation of the feature extractor.
* Through extensive experiments, we demonstrate the superior performance of our proposed method compared to other baseline approaches.

## Related Works

Head Initialization for Fine-TuningLP-FT, a two-stage transfer learning method, initializes the head layer with parameters obtained from LP prior to executing FT [22]. This highlights that random initialization of the head layer has the potential to cause feature distortion in FT, leading to degraded generalization performance. Nonetheless, the performance of LP-FT remains limited even with aligned head initialization, calling for further research on better head initialization methods. One promising technique is to stop the LP stage early before it reaches convergence, resulting in a non-converged head that improves performance during the subsequent FT stage [13]. Additionally, applying hardness-promoting augmentation during the LP stage can help mitigate feature distortion and also simplicity bias, leading to enhanced generalization performance [17]. Although these head initialization techniques have shown effectiveness in boosting the performance of the FT stage, they require a separate training stage to initialize the head layer before the FT stage. In contrast, our proposed DAFT integrates head adaptation seamlessly without the need for a distinct initialization stage.

Other Modifications for Fine-TuningThere are several other techniques to improve the performance of FT. Side-tuning involves fixing the feature extractor to preserve pretrained features and combining it with a small side network, then training only the side network and the head layer [14]. Other methods combine FT with the regularization of pre-trained parameters [23, 15, 16, 17], or the use of different learning rates for each layer [18, 19].

Figure 1: Distribution of similarity measures for three different fine-tuning methods. We fine-tune the pre-trained ResNet-50 model from MoCo-v2 [13] on fMoW [17] dataset. First, we extract features from the test data before applying each fine-tuning method. Then, after completing the fine-tuning process, we compute the similarity measures between the pre-fine-tuning and post-fine-tuning features for each method. The similarity measures, including cosine similarity and L2 distance, are computed on the fMoW test dataset. Notably, our DAFT exhibits the least distortion in the pre-trained features across both similarity measures.
et al., 2019; Clark et al., 2020). Our method falls under the category of employing different learning rates, as we use a larger learning rate for the head layer. However, unlike general methods that train the head layer with a learning rate approximately 10 times larger than that of the feature extractor, our method uses separate learning rates for the head layer and the feature extractor. This allows the head layer to align better with the ID dataset during the early stage of training and to converge with the adapting feature extractor.

Batch Normalization in Transfer LearningBatch normalization is a well-known technique for covariance shift reduction and better stable training Ioffe and Szegedy (2015). However, in transfer learning scenarios, where the data distribution between the source domain and target domain may significantly differ, the movement of statistics within batch normalization layers can result in severe feature distortion. As a solution, many methods choose to freeze the batch normalization statistics during fine-tuning to alleviate feature distortion Kumar et al. (2022); Ren et al. (2023); Trivedi et al. (2023). On the other hand, some methods leverage batch normalization with domain-specific statistics to encourage the learning of more generalized features Wang et al. (2019); Chang et al. (2019) or even use statistics from test batches to enhance robustness against domain shift Mirza et al. (2022); Lim et al. (2023). In contrast to existing approaches, our method aims to mitigate distortion of pre-trained features while still benefiting from batch normalization during training.

## Method

In this section, we introduce our approach, Domain-Aware Fine-Tuning (DAFT), which incorporates converting batch normalization and integrating LP and FT.

### Converting Batch Normalization

Batch Normalization (BN)The Batch Normalization (BN) technique Ioffe and Szegedy (2015) was introduced based on the observation that network training tends to converge faster when its inputs are whitened LeCun et al. (2002); Wiesler and Ney (2011). BN operates in two modes: training mode and test mode. In the training mode, it first computes batch statistics, \(\mu\) and \(\sigma^{2}\), for the input features \(z\). Then, scaling and shifting parameters, \(\gamma\) and \(\beta\), are applied to scale and shift the normalized values. The normalization process is represented as follows:

\[\hat{y}=\gamma\cdot\frac{z-\mu}{\sqrt{\sigma^{2}+\epsilon}}+\beta, \tag{1}\]

where \(\epsilon\) is a small constant for numerical stability. During training, BN computes the mean \(\mu\) and variance \(\sigma^{2}\) over a mini-batch of training data as

\[\mu=\frac{1}{m}\sum_{i=1}^{m}z_{i},\quad\sigma^{2}=\frac{1}{m}\sum_{i=1}^{m} {(z_{i}-\mu)^{2}}, \tag{2}\]

where \(m\) is the batch size. In the test mode, BN uses test statistics estimates, \(\mathrm{M}\) and \(\Sigma\), instead of \(\mu\) and \(\sigma\). Test statistics are obtained by using the moving averages of the train batch statistics. These moving averages are continuously updated during training to ensure consistent normalization during the test mode without the need for additional statistics computation.

BN Transfer IssueDue to the presence of _dataset bias_ or _domain shift_Quinnero-Candela et al. (2008), significant differences arise in the input distribution of batch normalization between the source and the target domains. Let \(\mathrm{M}s\) and \(\Sigma s\) respectively represent the test mean and test variance of batch normalization, pre-trained on the source domain. When training a pre-trained model on a new target domain, the values of \(\mathrm{M}_{s}\) and \(\Sigma_{s}\) undergo considerable updates, leading to a significant deviation from their original values. Moreover, the learning parameters also experience substantial adjustments during training to adapt and converge with the target domain statistics. These significant changes in batch normalization are one of the main factors that cause feature distortion during fine-tuning.

Batch Normalization ConversionTo mitigate this BN issue, we propose batch normalization conversion. Before starting the training process, we first compute batch statistics, \(\mu_{t}\) and \(\sigma_{t}\), for each mini-batch of the target training data. Next, we compute the new unbiased statistics, \(\mathrm{M}_{t}\) and \(\Sigma_{t}\), as follows:

\[\mathrm{M}_{t}=\mathrm{E}_{\mathcal{B}_{t}}[\mu_{t}],\quad\Sigma_{t}=\frac{m}{ m-1}\mathrm{E}_{\mathcal{B}_{t}}[\sigma_{t}^{2}], \tag{3}\]

where \(\mathcal{B}_{t}\) represents mini-batches of target training data. With these statistics estimated from the target domain, we can reformulate the batch normalization of the pre-trained model as follows:

\[\hat{y} =\gamma_{s}\cdot\frac{z-\mathrm{M}_{s}}{\sqrt{\Sigma_{s}+\epsilon }}+\beta_{s} \tag{4}\] \[=\gamma_{s}\cdot\frac{\sqrt{\Sigma_{t}+\epsilon}}{\sqrt{\Sigma_{s} +\epsilon}}\cdot\frac{z-\mathrm{M}_{t}+\mathrm{M}_{t}-\mathrm{M}_{s}}{\sqrt{ \Sigma_{t}+\epsilon}}+\beta_{s},\] \[=(\gamma_{s}\frac{\sqrt{\Sigma_{t}+\epsilon}}{\sqrt{\Sigma_{s}+ \epsilon}})\cdot\frac{z-\mathrm{M}_{t}}{\sqrt{\Sigma_{t}+\epsilon}}+(\beta_{s} +\gamma_{s}\frac{\mathrm{M}_{t}-\mathrm{M}_{s}}{\sqrt{\Sigma_{s}+\epsilon}})\] \[=\gamma_{t}\cdot\frac{z-\mathrm{M}_{t}}{\sqrt{\Sigma_{t}+\epsilon }}+\beta_{t},\]

where \(\gamma_{s}\) and \(\beta_{s}\) are the learning parameters of batch normalization that are pre-trained on the source domain. According to this reformulation, we define the new batch normalization parameters, \(\gamma_{t}\) and \(\beta_{t}\), with the target domain statistics as follows:

\[\gamma_{t}=\gamma_{s}\cdot\frac{\sqrt{\Sigma_{t}+\epsilon}}{\sqrt{\Sigma_{s}+ \epsilon}},\quad\beta_{t}=\beta_{s}+\gamma_{s}\cdot\frac{\mathrm{M}_{t}- \mathrm{M}_{s}}{\sqrt{\Sigma_{s}+\epsilon}}. \tag{5}\]

After computing these new parameters, we replace \(\gamma_{s}\) and \(\beta_{s}\) in the pre-trained model with \(\gamma_{t}\) and \(\beta_{t}\), respectively. Additionally, the statistics \(\mathrm{M}_{s}\) and \(\Sigma_{s}\) are replaced by \(\mathrm{M}_{t}\) and \(\Sigma_{t}\). After converting all batch normalization layers in the pre-trained model, we start training on the new target domain. With the converted BN parameters, the pre-trained model requires only slight adjustments to the batch statistics of the target domain. Furthermore, the movement of \(\mathrm{M}_{t}\) and
can be largely reduced since the batch statistics of the target training data are already close to \(\mathrm{M}_{t}\) and \(\Sigma_{t}\). As a result, the converted BN substantially mitigates feature distortion, leading to improved training performance on the target domain. The overall process of our BN conversion is summarized in Algorithm 1.

Utilizing our BN conversion technique, neural networks undergo more stable adjustments that aid in mitigating the distortion of pre-trained features, as shown in Figure 1. For a comprehensive understanding of how neural network evolves during adaptation, we compute the norm of the relative change using the equation:

\[\frac{\|\Delta W\|_{2}}{\|W\|_{2}}=\frac{\|\widetilde{W}-W\|_{2}}{\|W\|_{2}}, \tag{6}\]

where \(\widetilde{W}\) represents the parameter values after the training is completed, and \(W\) represents the initial parameter values before training begins. To conduct our evaluation, we employ a pre-trained ResNet-50 model from MoCo-v2 (Chen et al., 2020) and perform two types of training on fMoW (Christie et al., 2018) dataset: fine-tuning without our BN conversion (FT) and fine-tuning with our BN conversion (DAFT). For each case, we compute the norm of the relative change for all layers of the ResNet-50 model. It is important to note that \(W\) for FT refers to the value of the pre-trained model, whereas \(W\) for DAFT refers to the value after applying BN conversion to the pre-trained model. As illustrated in Figure 2, we can clearly observe that DAFT leads to significantly smaller relative changes compared to FT. This reduction is evident not only in the parameters of all layers but also in the statistics of all BN layers. This compelling evidence substantiates the effectiveness of our BN conversion technique, enabling the model to adapt more effectively to the target domain while inducing minor alterations.

Figure 2: Comparison of relative changes in learning parameters (2a, 2b, 2c) and BN statistics (2d, 2e) between FT and our DAFT on the fMoW dataset. We utilize the pre-trained ResNet-50 model from MoCo-v2 as the feature extractor and conduct each fine-tuning method. The ResNet-50 architecture consists of an Input Stem and 4 subsequent stages, with each stage indicated on the x-axis from left to right. ‘IS’ on the x-axis represents the Input Stem, and all layers within each stage are further indicated on the x-axis sequentially from left to right. The relative changes are computed between the initial values before each fine-tuning and the final values after each fine-tuning process. Note that the relative changes of BN statistics are represented in log scale.
### Integrating LP and FT

Our BN conversion technique can be applied to various transfer learning algorithms for neural networks that incorporate BN layers. However, considering the limitations of the recent LP-FT method, we integrate LP and FT in a single stage to leverage BN conversion more effectively.

#### 3.2.1 Limitation of LP-FT

Let \(\theta\) represent the parameters of the feature extractor denoted as \(f_{\theta}\), initialized with a pre-trained model. The classifier with parameters \(w\) is denoted as \(g_{w}\). While it is commonly a linear layer, it could also consist of multiple layers. In the LP-FT approach, the head classifier \(g_{w}\) is initialized using the value optimized by LP, and then FT is performed with a very small learning rate \(\eta\). As a result, the head layer undergoes only minor changes during FT and remains nearly unchanged after the FT stage. This implies that the head layer optimization primarily occurs during the LP stage, and it is not optimized in conjunction with the feature extractor's adaptation to the target domain. This limitation hinders performance, and it becomes particularly critical when there is a significant difference in data distribution between the source and target domains. To address this issue, we introduce integrating LP and FT into a single stage, allowing the head layer to be optimized with gradually adapting features.

#### 3.2.2 Integrated LP-FT

In order to integrate the LP and FT stages, we introduce separate learning rates, \(\eta_{\theta}\) and \(\eta_{w}\), for the feature extractor \(f_{\theta}\) and the head layer \(g_{w}\). While the conventional FT also suggests using a different learning rate for the head layer that is 10 times larger than that of the feature extractor, we independently determine the optimized learning rates for each component. To efficiently determine these optimized learning rates, we adopt a two-step procedure, sweeping over each learning rate. In the first step, we sweep over the learning rates \(\eta_{\theta}\) for the feature extractor with fixing \(\eta_{w}\). Having determined the optimized learning rate \(\eta_{\theta}\) for the feature extractor, we move on to the second step, where we sweep over the learning rates \(\eta_{w}\) while utilizing the previously determined optimized learning rate \(\eta_{\theta}\). By following this process, we integrate the FT and LP stages into a single approach, which distinguishes our approach from LP-FT. Moreover, this strategy significantly reduces the number of hyperparameter combinations and streamlines the search for proper learning rates.

Additionally, we employ a zero initialization strategy for the head layer. This approach is aimed at reducing the influence of gradients on the feature extractor during the initial training phases. By initializing the head layer with zeros, we try to avoid making significant changes to the feature extractor until the head layer has started to converge to a reasonable solution. However, in cases where the head layer comprises multiple layers, we opt for random initialization instead of zero values to enable gradient computation. Detailed distinctions between our DAFT and other fine-tuning techniques, including FT and LP-FT, are summarized in Table 1.

## 4 Experiments

In this section, we present the results of various experiments conducted to evaluate the effectiveness of our proposed method. We first present the experimental results on classification tasks and segmentation tasks. Additionally, we provide the results of additional experiments, including robustness evaluations and ablation studies.

### Experiments on Classification Task

#### 4.0.1 Dataset

We conduct experiments on various classification tasks and evaluate both In-Distribution (ID) and Out-of-Distribution (OOD) accuracy using the following datasets.

* **CIFAR-10**[10]: A dataset that contains 10 categories of objects. For the OOD dataset, we use two additional datasets, **CIFAR-10.1**[11] and **STL**[12]. CIFAR-10.1 is a subset of the TinyImage dataset [1] and has the same categories as CIFAR-10. STL has the same categories as CIFAR-10 except for'monkey' instead of 'frog'. Therefore, we remove the'monkey' class in STL to align with CIFAR-10 categories. For data augmentation, we resize the input image to 224x224 and apply random horizontal flip.
* **Entity-30** and **Living-17**[13]: Part of the BREEDS benchmarks, which are subpopulation shift datasets constructed using ImageNet. Entity-30 contains 30 categories of objects, and Living-17 contains 17 categories of animals. Each dataset consists of source and target domains. For both Entity-30 and Living-17, we use the source domain as the ID dataset and the target domain as the OOD dataset. Data augmentation involves RandomResizedCrop to 224x224 and random horizontal flip.
* **DomainNet**[14]: A dataset that includes 345 categories of common objects in six different domain including Clipart, Infograph, Painting, Quickdraw, Real, and Sketch. However, due to labeling noise in DomainNet, we used a subset of DomainNet [12] containing 40 categories of common objects from the Sketch, Real, Clipart, and Painting domains. Sketch domain is used for the ID dataset and the rest of the domains (Real, Clipart, Painting) are used for the OOD dataset. Data augmentation includes resizing the images to 256x256, random cropping to 224x224, and applying random horizontal flip.
* **fMoW**[10]: A remote sensing dataset that contains 62 categories of satellite images in five different regions including Asia, Europe, Africa, Americas, and Oceania. For our evaluation, we use images from the Americas region as the ID dataset, and images from the

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Method & Head initialization & LR for FT & BN mode & BN Conversion \\ \hline FT & Random & \(\eta_{\theta}=\eta_{w}\) & training mode & X \\ LP-FT & LP & \(\eta_{\theta}=\eta_{w}\) & test mode & X \\ DAFT & Zero & \(\eta_{\theta}\neq\eta_{w}\) & training mode & O \\ \hline \hline \end{tabular}
\end{table}
Table 1: Summary of details for our DAFT and other fine-tuning methods.
Europe and Africa regions are used for the OOD dataset.

As the fMoW data size is 224x224, we apply only random horizontal flip for data augmentation.

Pretrained ModelsTo ensure the generalizability of our method across different pre-trained models, we utilize three different models based on the ResNet-50 architecture: MoCo-v2 (Chen et al., 2020), CLIP (Radford et al., 2021), and SWAV (Caron et al., 2020). Exceptionally, for the fMoW dataset, we use MoCo-TP (Ayush et al., 2021) instead of MoCo-v2 as the pre-trained model.

Training DetailsConsistent with LP-FT (Kumar et al., 2022), we use \(\ell_{2}\)-regularized logistic regression classifier for linear probing and also choose the best \(\ell_{2}\)-regularization hyperparameter based on ID validation accuracy. For fine-tuning, we employ an SGD classifier with a cosine learning rate schedule and a batch size of 64. We fine-tune for 20 epochs on CIFAR-10, Entity-30, and Living-17, but extend the fine-tuning to 50 epochs on DomainNet and fMoW due to their limited number of images. Early stopping is applied based on the ID validation accuracy.

ResultsOur experimental results on five distinct datasets are summarized in Table 2. All of results are averaged over three separate runs. First, we observe that FT tends to exhibit superior performance in ID accuracy compared to LP, while LP tends to outperform FT in OOD accuracy. Across three diverse pre-trained models, our DAFT consistently achieves higher ID accuracy on all datasets in comparison to other baseline methods. Additionally, DAFT consistently exhibits higher OOD accuracy across most of the datasets.
ResultsTable 3 presents the results of our experiments on the segmentation task. Our DART demonstrates improved performance compared to other baseline methods for both the VOC and Cityscapes datasets. LP-FT also shows some enhancement by employing an LP-trained model, but it still achieves inferior performance compared to FT. These results show the significance of BN in the context of the segmentation task as well.

### Additional Experiments

RobustnessIn addition to ID and OOD test, we also evaluate the robustness of our method. After fine-tuning on the CIFAR-10 dataset, we evaluate the model's performance on the CIFAR-10-C dataset Hendrycks and Dietterich (2019). The CIFAR-10-C dataset comprises 15 types of corruptions grouped into four main categories: noise, blur, weather, and digital. Each corruption type is present at five different levels of severity. We calculate the Corruption Error for all levels of corruptions and then compute their average. The results of the robustness assessment are presented in Table 4. Interestingly, LP outperforms FT in terms of OOD accuracy, but it shows weaker performance in terms of robustness. And, our DART achieves lower average errors across various corruptions compared to other methods. Since this robustness experiment is conducted for the methods whose results are presented in Table 2, the Corruption Error values are also obtained as an average over three runs.

Ablation StudyTable 5 presents the results of our ablation study on BN conversion and integrated LP-FT. We conducted the study using a pre-trained ResNet-50 model from MoCo-v2 Chen et al. (2020) on the CIFAR-10 dataset. To evaluate the effectiveness of BN conversion alone, we applied it to each fine-tuning method, including FT, LP-FT, and integrated LP-FT. The results demonstrate that BN conversion is effective for most of OOD tests with all fine-tuning methods, indicating its importance in improving performance on OOD datasets. Additionally, we observe that integrated LP-FT without BN conversion performs similarly to LP-FT, revealing the limitations of the two-stage optimization and emphasizing the critical role of BN in the overall optimization process.

## Conclusion

In this paper, we introduce Domain-Aware Fine-Tuning (DAFT), a novel approach designed to enhance the adaptability and performance of fine-tuned neural networks. Our method optimizes performance and minimizes network modification by aligning batch normalization layers with the target domain. Additionally, the integration of LP and FT allows the head layer to be optimized with gradually adapting features. The widespread use of batch normalization layers in many practical networks makes DAFT a valuable solution for real-world applications. Overall, DAFT bridges the gap between pre-trained models and new target domains, which contributes to improved model performance and generalization.

\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{Model} & \multirow{2}{*}{Method} & \multicolumn{3}{c}{Noise} & \multicolumn{3}{c}{Blur} & \multicolumn{3}{c}{Weather} & \multicolumn{3}{c}{Digital} & \multicolumn{3}{c}{mCE} \\ \cline{3-14}  & & Gauss. & Shot & Impulse & Defocus & Glass & Motion & Zoom & Snow & Frost & Fog & Brightness & Contrast & Elastic & Pixel & JPEG \\ \hline \multirow{6}{*}{MoCo} & LP & 47.54 & 40.05 & 54.24 & 19.48 & 50.79 & 33.38 & 23.50 & 22.80 & 23.57 & 23.72 & 10.12 & 18.26 & 28.71 & 30.48 & 25.65 & 30.15 \\  & FT & 56.07 & 41.87 & **45.48** & 9.88 & 40.95 & 17.10 & 12.42 & 9.33 & 11.91 & 7.40 & 3.24 & 9.83 & **12.62** & 27.17 & 20.89 & 21.74 \\  & LP-FT & 46.50 & 36.01 & 48.30 & 8.11 & **38.19** & **14.56** & **10.04** & **9.13** & 11.27 & **7.10** & 3.26 & 10.99 & 12.89 & **24.56** & **19.20** & 20.03 \\  & DAFT & **37.43** & **28.84** & 46.72 & **8.01** & 39.79 & 16.26 & 10.84 & 9.32 & **10.85** & 8.15 & **3.22** & **9.12** & 14.40 & 28.54 & 19.47 & **19.40** \\ \cline{2-14}  & LP & 65.06 & 57.71 & 56.24 & 24.49 & 57.37 & 35.31 & 27.78 & 26.50 & 27.89 & 23.44 & 15.42 & 31.82 & 30.06 & 28.48 & 33.82 & 36.09 \\  & FT & **46.54** & **35.05** & **37.17** & 17.59 & 52.15 & 24.75 & 22.03 & 19.57 & 22.23 & 12.68 & 8.41 & 24.71 & 18.95 & **27.64** & **18.74** & 25.88 \\  & LP-FT & 57.33 & 46.14 & 48.53 & 14.40 & 51.64 & 24.41 & 18.31 & 15.76 & 18.24 & 11.63 & 8.05 & 23.77 & 20.39 & 33.87 & 27.27 & 27.98 \\  & DAFT & 61.58 & 48.10 & 40.71 & **13.55** & **46.95** & **21.96** & **16.87** & **13.78** & **15.88** & **9.74** & **6.15** & **18.39** & **17.61** & 31.95 & 22.99 & **25.75** \\ \hline \multirow{6}{*}{SWAV} & LP & 49.95 & 42.77 & 60.15 & 17.42 & 45.79 & 27.20 & 20.22 & 19.31 & 22.27 & 19.62 & 8.58 & 12.46 & 25.01 & 27.84 & 25.17 & 28.25 \\  & FT & 51.71 & 38.55 & 46.94 & 10.05 & 39.13 & 16.83 & 11.45 & **8.74** & 10.62 & 7.68 & 3.57 & 10.89 & **12.36** & **29.21** & 20.96 & 21.25 \\ \cline{1-1}  & LP-FT & 50.05 & 37.70 & **36.61** & **8.72** & **33.36** & **15.41** & **10.44** & 8.84 & 11.08 & 6.57 & 3.43 & 9.36 & 12.54 & **25.27** & 20.50 & 19.37 \\ \cline{1-1}  & DAFT & **41.39** & **30.18** & 46.90 & 8.86 & 36.13 & 15.98 & 10.87 & 8.90 & **9.99** & 7.60 & **3.12** & **7.69** & 12.82 & 25.39 & **19.45** & **19.02** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Corruption Error (%) for 15 types of corruptions. The mean Corruption Error (mCE) is calculated for all corruptions. A lower value of Corruption Error indicates better performance. Best results are highlighted in bold.

\begin{table}
\begin{tabular}{c l l l} \hline \hline \multirow{2}{*}{Model} & \multirow{2}{*}{Method} & \multicolumn{3}{c}{CIFAR-10} \\ \cline{3-4}  & & ID & CIFAR-10.1 & STL \\ \hline \hline \multirow{6}{*}{MoCo} & FT & 97.66 & 92.95 & 81.18 \\  & + BN conversion & 97.59 & 93.27 & 84.05 \\ \cline{2-4}  & LP-FT & 97.66 & 93.60 \\  & + BN conversion & 97.31 & 93.00 & 91.66 \\ \cline{2-4}  & Integrated LP-FT & 97.63 \\  & + BN conversion & 97.84 & 94.18 & 92.36 \\ \hline \multirow{6}{*}{CLIP} & FT & 93.88 \\  & + BN conversion & 95.40 \\ \cline{2-4}  & LP-FT & 94.02 \\ \cline{2-4}  & + BN conversion & 94.70 \\ \cline{2-4}  & Integrated LP-FT & 94.29 \\  & + BN conversion & 95.49 \\ \hline \multirow{6}{*}{SWAV} & FT & 97.55 \\  & + BN conversion & 97.62 \\ \cline{2-4}  & LP-FT & 97.50 \\ \cline{1-1}  & + BN conversion & 97.61 \\ \cline{1-1}  & Integrated LP-FT & 97.28 \\ \cline{1-1}  & + BN conversion & 97.77 \\ \hline \hline \end{tabular}
\end{table}
Table 5: BN conversion effect for each fine-tuning method.