CLDA: Collaborative Learning for Enhanced Unsupervised Domain Adaptation
Minhee Cho Hyesong Choi Hayeon Jo Dongbo Min


A Data Selection Approach for Enhancing Low Resource Machine Translation Using Cross-Lingual Sentence Representations



    
Nidhi Kowtal * 

SCTR's Pune Institute of Computer Technology

Pune, India 

kowtalnidhi@gmail.com

Tejas Deshpande * 

SCTR's Pune Institute of Computer Technology

Pune, India 

tejasdeshpande1112@gmail.com
 

Raviraj Joshi

Indian Institute of Technology Madras, India

L3Cube Labs, Pune 

Pune, India 

ravirajoshi@gmail.com

    September 9, 2024
==============================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 



Unsupervised Domain Adaptation (UDA) endeavors to bridge the gap between a model trained on a labeled source domain and its deployment in an unlabeled target domain. However, current high-performance models demand significant resources, resulting in prohibitive deployment costs and highlighting the need for small yet effective models. For UDA of lightweight models, Knowledge Distillation (KD) in a Teacher-Student framework can be a common approach, but we find that domain shift in UDA leads to a significant increase in non-salient parameters in the teacher model, degrading model's generalization ability and transferring misleading information to the student model. Interestingly, we observed that this phenomenon occurs considerably less in the student model. Driven by this insight, we introduce Collaborative Learning, a method that updates the teacher's non-salient parameters using the student model and at the same time enhance the student's performance using the updated teacher model. Experiments across various tasks and datasets show consistent performance improvements for both student and teacher models. For example, in semantic segmentation, CLDA achieves an improvement of +0.7% mIoU for teacher and +1.4% mIoU for student compared to the baseline model in the GTA to Cityscapes. In the Synthia to Cityscapes, it achieves an improvement of +0.8% mIoU for teacher and +2.0% mIoU for student.


 






§ 1. INTRODUCTION




While the availability of extensive labeled datasets has driven remarkable advancements in various computer vision tasks, there exists a much larger quantity of unlabeled data in real-world scenarios. To accommodate a variety of real-world applications, it is crucial for deep neural networks (DNNs) to generalize to these unlabeled data. However, generalizing the models trained in labeled data (source domain) to unlabeled data (target domain) is non-trivial and pose several challenges. Unsupervised domain adaptation (UDA) has been studied to resolve this issue, which transfers knowledge learned from the source domain to the target domain through adversarial training <cit.> or self-training <cit.>.




Although UDA methods have made remarkable progress in recent years, the best-performing approaches are mostly based on resource-intensive networks, such as DAFormer <cit.> and HRDA <cit.>. Deploying such models in resource-constrained environments is often infeasible. Training compact models in the same way using existing UDA approaches is not very effective either, leading to the necessity of developing compact yet effective models. One promising solution is to apply Knowledge Distillation (KD) <cit.> that leverages a high-capacity teacher model to train a compact student model. Fig. <ref> (a) shows the example of applying KD for training a lightweight student model in the UDA setup <cit.>. The student model is trained by leveraging knowledge from a fixed teacher model that possesses domain adaptation capabilities against the target domain by utilizing adversarial learning <cit.> to minimize the discrepancies between the source and target domains.




















However, directly leveraging the fixed teacher model in the UDA problem exhibits inherent limitations by heterogeneous data distributions of source and target domains. In other words, due to the gap between the data distributions of the source and target domains, some of teacher parameters learned on the source domain may not be useful or even adversely work on the target domain <cit.>. This discrepancy generates non-salient parameters, limiting  generalization ability <cit.> when used in the target domain. Consequently, the misleading information from the frozen teacher model can be transferred to the student model in Fig. <ref> (a). We define this as the problem of Domain Shift induced Non-salient parameters (DSN), which will be detailed in the following analysis section.





We tackle this challenging problem from a new perspective of updating the teacher model to alleviate the DSN problem, unlike KD approaches that use fixed teacher models in Fig. <ref> (a). 
To this end, we first analyzed the occurrence of the DSN problem across layers in both the teacher and student models through Layer Saliency Rate (LSR), which indicates the contribution of each layer to the model performance when the layer is removed. 
Our key observation from Fig. <ref> is that approximately half of the layers in the teacher model, marked as `Non-salient layer', suffer from the DSN problem, while less than 10% of the layers in the student model are affected. This indicates that the student model contributes more effectively to the target domain, and thus can effectively complement the DSN problem of the teacher model. Going one step further, we introduce collaborative learning, where the knowledge of the improved teacher model is used to train the student model through the distillation process, as depicted in Fig. <ref> (b).























 A naive way to improve the teacher model would involve retraining the layers of the teacher model that exhibit DSN problem to resemble the student model. However, simultaneously training both the teacher and student models requires significant computational costs and  large memory capacity, especially for heavy teacher models. Instead, as illustrated in Fig. <ref> (b), we attempt to enhance the performance through mutual information exchange without the need for backpropagation to the teacher model, facilitating knowledge transfer from teacher to student (T → S) and vice versa (S → T). To achieve this, it is necessary to carefully analyze the interaction that occurs when information is transferred in both directions. 
Initially, we observed that when knowledge is transferred as ‘T → S’, certain parts of the student model are more influenced by the teacher compared to an independently trained student model. Based on this, it is reasonable to update the teacher model using the student model (S → T) in the regions of active interaction occur. Consequently, we explored strategies to identify and leverage the appropriate relationship between the student and teacher models in these areas to enhance the teacher model's performance. Further details are discussed in section 4.3.


























Based on this comprehensive analysis, we introduce Collaborative Learning for UDA, termed CLDA, a novel approach that enables the teacher and student models to complement each other, simultaneously enhancing the performance of both models. To maximize the synergy between the two models, we transfer knowledge from the teacher model to the student model (T → S), and simultaneously identify the appropriate layers in the student model that correspond to the non-salient layers in the teacher model. Then, the non-salient layer of the teacher is updated based on the corresponding layer of the student (S → T). We applied CLDA to semantic segmentation and classification tasks, conducting experiments across various datasets and domains. Consistent performance improvements were observed, demonstrating the effectiveness of our approach. Notably, our approach achieved performance comparable to the baseline teacher model for the lightweight student model. 
To summarize, our key contributions include the followings:



 
  * We address the problem of Domain Shift Induced Non-salient parameters (DSN) in UDA and introduce a novel method to effectively resolve this issue.


  * Through detailed analysis, we discover the student model can sufficiently complement the DSN problem that occurs in the teacher model.


  * We propose a novel and effective strategy for UDA that enables the teacher and student models to complement each other, thereby simultaneously enhancing the performance of both, without requiring backpropagation in the teacher model.


  * Experimental results on several datasets and models show consistent performance improvements in both the teacher and student models.





§ 2. RELATED WORK




  §.§.§ Unsupervised Domain Adaptation


To effectively adapt a network to the target domain, several strategies have been proposed, most can be categorized into adversarial learning <cit.> and self-training <cit.>. Inspired by the success of Generative Adversarial Networks (GANs) <cit.>, adversarial learning methods aim to learn invariant representations that reduce the distance between source and target distributions at the image <cit.>, feature <cit.>, and output levels<cit.>. Recently, self-training has emerged as a promising alternative for domain adaptation. Self-training leverages pseudo labels <cit.> for unlabeled target data<cit.>. To mitigate the noise in pseudo labels caused by domain shift, approaches such as confidence thresholding <cit.>, prototypes <cit.>, and data augmentation <cit.> have been employed. 




  §.§.§ Knowledge distillation
 
Knowledge distillation (KD) aims to transfer the knowledge <cit.> acquired by a complex teacher model <cit.> to a smaller student model. Since <cit.> introduced the concept of knowledge distillation, where the teacher model’s dark knowledge is provided through temperature-scaled softmax outputs, various studies have been inspired to utilize more teacher information <cit.>. Some methods have been proposed to enhance interaction between the teacher and the student <cit.>. For instance, <cit.> proposed a novel student-friendly learning technique within the teacher network to facilitate knowledge distillation. However, these methods assume the teacher and student operate within the same domain. When this assumption fails, the teacher's performance degrades, transferring incorrect information to the student. Our research proposes a KD method that remains robust despite domain shifts.







§ 3. PRELIMINARIES

UDA aims to alleviate the performance degradation caused by a domain gap between source domain datasets P = { ( x^i_p, y^i_p  )}^N_p_i=1 and target domain datasets Q = { ( x^i_q  )}^N_q_i=1, where N_p and N_q indicate the number of training images in the source and target domains. The model f comprises a feature extractor h_ϕ parameterized by ϕ and a head g_θ parameterized θ, i.e. f_θ ,ϕ = g_θ ( h_ϕ ( x  )  ).


The training process employs a total loss ℒ=ℒ_sup+ℒ_da consisting of a supervised loss ℒ_sup using the source data P and an domain adaptation loss ℒ_da using the target data Q. The type of the supervised loss ℒ_sup varies depending on tasks <cit.>. Also, adaptation loss L_da is defined according to the UDA strategy such as adversarial learning <cit.> or self-training <cit.>. The overall training process is as follows:


    min_θ,ϕ1/N_P∑_k=1^N_Pℒ_sup^k + 1/N_Q∑_k=1^N_Qℒ_da^k .













§ 4. ANALYSIS




To address the DSN problem in UDA, we investigate the characteristics of the teacher and student models <cit.>. Given that a Transformer-based method has recently outperformed previous CNN-based UDA methods across various benchmark datasets under the UDA setting, we assume the Transformer-based framework <cit.> throughout this paper. Key findings from our analysis are summarized as follows:


     
  * Layer Saliency Analysis: By analyzing Layer Saliency Rate (LSR), we found that the occurrence of DSN at the layer level was significantly lower in the student model compared to the teacher model. This indicates that the layers of the student model contribute more substantially to the target domain.
    
  * Layer Contribution: We discovered that the student model contributes approximately 2.5 times more to the target domain than the teacher model. Consequently, we conclude that the student model can effectively complement the teacher model in addressing the DSN problem. 
    
  * Analysis for Collaborative Learning: Knowledge transfer from the teacher to the student (T → S) significantly influences the attention module in the deeper layers. Also, when updating the teacher from the student (S → T), a channel similarity-based approach proves to be effective.













 §.§ 4.1. Layer Saliency Analysis


To evaluate the occurrence of DSN problem at the layer level within the model, we investigate the distribution of saliency at the layer level. For this purpose, we define a Layer Saliency Rate, referred to as LSR, to measure the saliency of each layer.   Given a domain-adapted model, the LSR for the weight ϕ at layer i within the task accuracy R for the target domain is defined as follows: 


    LSR(f,ϕ ^(i)) = | R(f(ϕ ))-R(f(ϕ -ϕ _i)) |  .


The LSR value indicates the contribution of a layer to the model's performance when the layer i is removed. A higher value means a higher contribution of the layer. For semantic segmentation, we designate layers with LSR values less than 0.1% as non-salient layers, and for classification, we designate layers with LSR values less than 0.3% as non-salient layers. Conversely, layers in the opposite case are considered salient. Fig. <ref> reveals that while the non-salient layers occupy a large portion of the teacher model, the student model exhibits the opposite result. In the layers of the student model, the proportion of salinet layers is overwhelmingly high.
In the case of DAFormer <cit.>, these non-salient layers comprise more than half of the teacher model but are not found in the student model. This indicates that the proportion of layers contributing to the target domain is significantly higher in the student model.




 §.§ 4.2. Layer Contribution



To explore the degree of contribution of the student model to the target domain relative to the teacher model, we employ a popular method for measuring similarity in neural networks; Centered Kernel Alignment (CKA) <cit.>. Given two representations, X and Y of the same inputs,  CKA(X,Y) ∈ [0,1] indicates how similar (close to 1) or dissimilar (close to 0) the two are. We measure the CKA similarity for every pair of modules in the layer X ∈ℝ^n × p_1 and Y ∈ℝ^n × p_2, within the teacher and student models, where the modules with p_1 and p_2 neurons are evaluated n examples in the target domain. Fig. <ref> shows the results as a heatmap. We observed that the lower half of the student model's layers functionally correspond to twice the number of the teacher model's layers. The remaining half of the student model's layers functionally align with 2.5 times the number of the teacher model's layers. These percentages indicate that the student's layer contributes significantly to the target, enough to  resolve the DSN problems originating from the teacher's layer.






 §.§ 4.3. Analysis for Collaborative Learning






 To address the DSN problem in the teacher model, it is crucial to effectively leverage the layers of the student model, which are less affected by DSN problem. Understanding the information exchange and interaction between the two models is essential for this purpose. With this understanding, the teacher model can effectively incorporate the strengths of the student model, ultimately deriving the optimal solution to the DSN problem. Based on this understanding, we propose a novel Collaborative Learning approach that exploit the strengths of both models.




  §.§.§ Teacher to Student

Initially, we investigate the changes when knowledge was transferred from the teacher model to the student model. Our hypothesis posited that when the teacher model exerts greater influence on certain modules of the student model, notable changes in the parameters of those modules are observed compared to a student model trained independently. To validate this, we measured the average per-pixel variation rate (PVR) across modules within the layer, such as attention module and mlp module, between the distilled student model f_ID and the independently trained student model f_K as follows:

    PVR (f,ϕ^(m) ) = ∑_d = 0^D| ϕ_ID,d^m - ϕ_K,d^m|/D .



where ϕ^i,m denotes the weight for module m within layer i, D represents the dimensions of module m.  In Fig. <ref>, we identified two key findings. First, the Attention module was found to be more affected than the MLP module. 
Second, the deeper layers of the model exhibited more pronounced effects. These results suggest a need to design collaborative learning strategies particularly in the Attention modules and deeper layers, where the teacher's knowledge has substantial impact.








  §.§.§ Student to Teacher

 Based on the aforementioned observations, it is a reasonable choice to update the teacher model by leveraging the student model in areas where active interaction occurs. To determine the optimal layer in the student's attention module that corresponds to the teacher model, we carefully considered three key aspects: 1) Token-wise relationships, 2) Channel-wise relationships, and 3) Combining both 1 and 2. We measure each relationship as cosine similarity for the attention feature map in the student's layer based on a randomly selected non-salienct layer of the teacher. As shown in Fig <ref> (b), examining the token-wise relationships reveals that most layers exhibit low similarity and inconsistent similarity distributions. In contrast, the channel-wise relationships (Fig <ref>a) show a consistent distribution.
Detailed experimental results can be found in Section 6.2.
This findings leads to the following design: In the attention modules of the deeper layers, it is crucial to consider the channel similarity between the student and teacher models ,during updating the teacher model from the student model (S → T).






§ 5. METHOD




 §.§ 5.1 Motivation and Overview


Recent advancements in knowledge distillation have primarily utilized a frozen teacher model to transfer information to the student. However, our investigations reveal that this static approach is not appropriate for UDA due to the DSN problem in the teacher, as illustrated in Fig. <ref>. Consequently, this inadequacy diminishes the teacher model's generalization ability, which in turn increases the risk of spreading incorrect information to the student.
Intriguingly, our analysis indicates that the student has the capability to address these issues. This observation prompts us to propose a novel approach; collaborative learning between the teacher and the student. The proposed CLDA consists of the following stages.

Stage 1: In order to boost a synergy between these two models, the KD is first conducted from the teacher model to the student model (T → S).

Stage 2: A layer mapping process identifies the student layers used to update the non-salient layers of the teacher model.

Stage 3: The non-salient layer of the teacher is updated based on the corresponding layer of the student (S → T).

The proposed CLDA framework is illustrated in Fig. <ref>.








 §.§ 5.2 CLDA framework



  §.§.§ Stage 1: Knowledge Distillation (T → S)

The training process consists of two stages. In the first stage, the baseline model of substantial size is trained following the same DA procedure as described in <cit.>. Subsequently, this model, possessed with domain adaptation capabilities, serves as the teacher model f_T in our framework. The distillation process is computed in the output space using the ℒ_distill :

    ℒ_distill^(i) = - ∑_j=1^H × W∑_c=1^N_C q^ip̂_T^(i,j,c)log f_S(x_q^i)_(j,c) .


where H and W are height and width of an target image x_q, respectively, while N_C is the number of categories shared between the source and the target domain. Hard labels p̂_T are generated from the teacher's most confident class prediction.

    p̂_T^(i,j,c) = [ c = max_c' f_T(x_q^(i))^(j,c')] .


where [  · ] denotes the Iverson bracket. As the hard-labels are potentially wrong, the loss is weighted by the quality estimate q <cit.>.


















  §.§.§ Stage 2: Layer Mapping


 Within the teacher's blocks, non-salient layers are identified using Eq.(<ref>), and subsequently, 30% of these non-salient layers are randomly selected for collaborative learning. To effectively mitigate the non-salient layers of the teacher model, it is crucial to accurately identify the corresponding student layers. Based on our analysis, we evaluate the channel relationships between the teacher and student within an attention module. Therefore, we extract the attention feature map A_T,γ∈ℝ^B × N × C from the target image at a non-salient layer γ from the teacher model f_T, where B and N are the batch size and the number of tokens, respectively, while C is the number of channels. Similarly, we extract attention feature maps {A_S,i}_i=1^L_s∈ℝ^B × N × C from the same target image across all layers L_s within the block of the student model.  We then calculate the cosine similarity score ϑ between the channels within the attention feature maps of the teacher and student models in training step η.
 
 
    ϑ^η_i = ∑_j=1^CA_T,γ^n[:, : , j] · A_S,i^n[:, : , j]/|A_T,γ^n[:, : , j]| · |A_S,i^n[:, : , j]| .

 Finally, the student's layer with the highest similarity is selected to update the teacher’s non-salient layer.

    i^* = iargmax∑_η=T_2^T_3 ϑ^η_i .






  §.§.§ Stage 3: Teacher Update (S → T) 



By adopting an Exponentially Moving Average (EMA) approach, we can gradually update the non-salient parameters of the teacher model using the parameter of the student model, thereby avoiding additional backpropagation in the teacher model. After each training step η, the teacher
model f_T is updated with the exponentially moving average of the weights of the student f_S:



    ϕ_T,γ^η←αϕ_T,γ^η + (1 - α) ϕ_S,i^*^η .



The total loss of the proposed CLDA is formulated as

    L_CLDA =  min_θ,ϕ1/N_P∑_k=1^N_Pℒ^k_sup + 1/N_Q∑_k=1^N_Qℒ^k_distill .


The overall algorithm is summarized in Algorithm 1.























        



















  





















































§ 6. EXPERIMENTS









In this chapter, we demonstrate experimental results under different settings in order to verify the effectiveness of our proposed method.
We compare our method with other existing baselines on different benchmarks in Sec. 6.1. We also analyze the contribution from each of our vital model components in ablation studies in Sec. 6.2. Detailed information regarding the experiments and classification performance can be found in the supplementary material.



 §.§ 6.1. CLDA for semantic segmentation


First, we integrated CLDA with various Unsupervised Domain Adaptation (UDA) methods. Tab. <ref> and <ref> demonstrate that CLDA consistently enhances performance across different UDA methods and datasets, confirming improvements in both the teacher and student models. Specifically, when employing DAFormer on the GTA-to-Cityscapes dataset, the teacher model exhibited a performance improvement of  0.7%, while the student model showed an enhancement of 1.4%. In addition, using HRDA, the teacher and student models achieved performance improvements of 0.8% and 0.7%, respectively. Similarly, on the Synthia-to-Cityscapes dataset, CLDA improved performance by 0.8% for the teacher and 2.0% for the student with DAFormer, and by 0.3% for the teacher and 1.5% for the student with HRDA. These findings suggest that CLDA can be flexibly combined with various Transformer-based models, notably enhancing the teacher model's performance without calculating the backpropagation of the teacher. Thereby, effectively transmitting informative knowledge to the student model and maximizing the synergy between the two models.









































 §.§ 6.2. Ablation study of CLDA



  §.§.§ Component Ablation: 
We conducted a component ablation study to gain further insights into CLDA by incrementally removing its components and evaluating performance on the GTA→Cityscape dataset using DAFormer[MiT-b3] . In Tab. <ref>. The complete CLDA configuration achieves a mIoU of 69.5 (row 2), which is +1.4% mIoU higher than DAFormer (row 1).
When only distillation from the teacher to the student (S → T) was applied , the performance showed a 0.8% improvement over the DAFormer. However, when updating the student to teacher layers (T → S) without a mapping process, the mismatch between student and teacher layers led to a performance degradation of 0.6%, compared to the initial (S → T) approach. On the other hand, when Layer mapping was performed before updating, an improvement of 0.6% was observed. These findings highlight the importance of accurately matching the student's layer for the teacher's non-salient layer during CLDA training.




  §.§.§ Mapping Ablation: 
In the previous chapter, we analyzed the relationship between the student and teacher models from three perspectives: 1) token-wise, 2) channel-wise, and 3) a combination of both. To gain deeper insights, we conducted experiments on the GTA to Cityscapes with DAFormer[MiT-b3], in Tab. <ref>. When measuring the similarity of the attention feature maps between the two models at the token level (Row 1), we observed a performance decrease of 0.5% compared to the scenario where knowledge was transferred from teacher to student. This decline is attributed to the difficulty of identifying the appropriate student layer for updating, as token-wise approach failed to capture the correct relational context. Conversely, the channel-wise approach (row2) led to the update of suitable layers, resulting in a performance improvement of 1.4% over the DAFormer. When considering both token-wise and channel-wise relationships by flattening the attention feature maps, the performance improved compared to the token-wise approach but did not surpass the results achieved with channel-wise approach, indicating difficulty in identifying the optimal layers for update.



























































§ 7. CONCLUSION

We explore the underexamined yet practical challenge of developing compact and efficient models in DA. We analyze the DSN issue caused by a fixed teacher model in conventional KD within DA. Based on this analysis, we propose CLDA, a practical approach that addresses the DSN problem by leveraging the complementary effects between teacher and student models. A notable strength of CLDA is its ability to resolve the DSN issue without additional backpropagation for the teacher model. Our empirical results demonstrate significant performance gains for both teacher and student models across various settings. Future work could include a unified theoretical analysis of the DSN problem.
Limitations. CLDA demonstrates effectiveness across various environments, but its reliance on the teacher model may introduce constraints in some scenarios. This suggests potential for integrating additional techniques to enhance generalization without compromising the model’s efficiency.






