Non-target Divergence Hypothesis: Toward Understanding Domain Gaps in Cross-Modal Knowledge Distillation
Yilong Chen Zongyi Xu Xiaoshui Huang Shanshan Zhao Xinqi Jiang Xinyu Gao Xinbo Gao


















































IEEE TRANSACTIONS ON MULTIMEDIA
Shell et al.: Bare Demo of IEEEtran.cls for IEEE Journals


























Non-target Divergence Hypothesis: Toward Understanding Domain Gaps in Cross-Modal Knowledge Distillation
    Yilong Chen,
        Zongyi Xu,
        Xiaoshui Huang,
        Shanshan Zhao,
        Xinqi Jiang,
        Xinyu Gao,
        and Xinbo Gao, Fellow, IEEE
    September 9, 2024
==============================================================================================================================================================





§ ABSTRACT 
Compared to single-modal knowledge distillation, cross-modal knowledge distillation faces more severe challenges due to domain gaps between modalities. Although various methods have proposed various solutions to overcome these challenges, there is still limited research on how domain gaps affect cross-modal knowledge distillation. This paper provides an in-depth analysis and evaluation of this issue. We first introduce the Non-Target Divergence Hypothesis (NTDH) to reveal the impact of domain gaps on cross-modal knowledge distillation. Our key finding is that domain gaps between modalities lead to distribution differences in non-target classes, and the smaller these differences, the better the performance of cross-modal knowledge distillation. Subsequently, based on Vapnik-Chervonenkis (VC) theory, we derive the upper and lower bounds of the approximation error for cross-modal knowledge distillation, thereby theoretically validating the NTDH. Finally, experiments on five cross-modal datasets further confirm the validity, generalisability, and applicability of the NTDH.





Cross-Modal Knowledge Distillation, Domain Gaps, Multimodal Fusion.

















§ INTRODUCTION




















In recent years, cross-modal knowledge distillation (KD) has expanded the traditional KD approach to encompass multimodal learning, achieving notable success in various applications <cit.>. However, when there are considerable domain gaps in cross-modal KD, even a more accurate teacher model may not effectively instruct the student model. To overcome these challenges, many researchers have sought to enhance the effectiveness of cross-modal KD by designing efficient fusion strategies <cit.> or developing novel loss functions <cit.>. However, most of these methods focus on complex technical designs with less emphasis on exploring their theoretical foundations, which is the focus of this paper. 


Xue et al. <cit.> are the first to theoretically focus on KD under modality differences, proposing the Modality Focus Hypothesis (MFH). This hypothesis posits that the performance of cross-modal KD hinges on the modality-general decisive features preserved in the teacher network. These features indicate the degree of alignment between different modalities, with greater alignment leading to improved KD outcomes. Fig. <ref>(a) shows an example of a multimodal dataset with both audio and images, where the image data includes not only the scene of guitar music but also background information, leading to incomplete modality alignment; according to the MFH, improved feature alignment is expected to enhance cross-modal KD effectiveness. However, the above hypothesis has two shortcomings: (1) It cannot explain why cross-modal KD might still fail even when there is modality alignment, as shown in situations like those in Fig. <ref>(b)-(d). (2) It lacks a mathematical definition of modality-general key features, making it difficult to identify these features in practical applications.



To address these deficiencies, we first pre-register or align the modalities involved in our study to eliminate the effects caused by unregistered modalities, thereby focusing our research on domain gaps. We adopt this approach because we believe that misalignment is merely an external manifestation, while the actual domain gaps between modalities are the core distinction between cross-modal and single-modal KD.

Secondly, we clearly define the key factors affecting cross-modal KD. Specifically, inspired by <cit.>, we divide classification predictions into two levels: (1) Target class prediction distribution: a binary prediction distribution for the target category and all non-target categories; (2) Non-target class prediction distribution: a multi-category prediction distribution for each non-target category. We find that for cross-modal KD, the distribution divergence in non-target categories is the decisive factor. The smaller the divergences, the better the effect of cross-modal KD, which we refer to as the Non-target Divergence Hypothesis. Since the distance of non-target distributions can be easily measured using existing distance functions, they can be explicitly defined and calculated. For example, in multimodal point cloud semantic segmentation, both point clouds and images classify road areas. These predictions can be divided into road class prediction distribution and non-road class prediction distribution. If the distribution difference of the non-road class predictions is large, the effectiveness of cross-modal KD is significantly affected and may even fail, as shown in Fig. <ref>. The purpose of this work is to prove the validity of the hypothesis from a theoretical analysis perspective. Our major contributions are the following:







  
  1)
  We propose the Non-target Divergence Hypothesis (NTDH), which posits that the divergence of non-target distributions between modalities is a key factor determining the effectiveness of cross-modal KD.
  
  2)
  We theoretically prove the upper and lower bounds of the cross-modal KD approximation error, thereby validating the rationality of NTDH.
  
  3)
  We design a weight regulation method and a masking method to experimentally verify the hypothesis. Experimental results on five multimodal datasets support the proposed NTDH.
 





§ RELATED WORK




 §.§ Unimodal KD


KD is a general technique for transferring knowledge from a teacher network to a student network, widely applied in various vision tasks <cit.>. Although progress has been made in improving distillation techniques and exploring new application areas, research on the mechanisms underlying KD remains limited <cit.>. For example, Cho et al. <cit.> and Mirzadeh et al. <cit.> investigate KD in the context of model compression, focusing on the performance of student and teacher networks when their model sizes differ. They point out that a mismatch in capacity between the student and teacher networks can lead to KD failure. Ren et al. <cit.> analyze KD in vision transformers and find that the inductive bias of the teacher model is more important than its accuracy in improving the performance of the student model. However, the aforementioned methods mainly focus on the factors affecting unimodal KD, while research on cross-modal KD remains limited.










 §.§ Crossmodal KD



With the widespread use of the internet and the increasing application of multimodal sensors, there is a surge of interest in multimodal learning <cit.>. In line with this trend, KD extends to cross-modal KD, which commonly applies in three scenarios. First, Knowledge Expansion: Cross-modal KD compensates for insufficient data in a single modality. For instance, Ahmad et al. <cit.> use multimodal MRI data to overcome clinical data scarcity, while Li et al. <cit.> transfer RGB-trained models to infrared data, addressing its data limitations. Similar approaches include using Galvanic Skin Response signals to offset Electroencephalogram data collection challenges <cit.>, supplementing Synthetic Aperture Radar images with RGB data <cit.>, and transferring English-trained models to other languages <cit.>. Second, Multimodal Knowledge Fusion: Cross-modal KD fuses complementary information from different modalities into a unimodal network, requiring multimodal data only during training and simplifying deployment <cit.>, such as using Lidar for RGB images <cit.>, RGB for point clouds <cit.>, facial images for voice data <cit.>, and RGB to enhance thermal imaging <cit.>. Large language models also aid in cross-modal retrieval <cit.>. Third, Additional Constraints: In some cases, significant differences in network structures for different modalities make feature layer fusion highly challenging. The loss function of cross-modal knowledge distillation then acts as a regularization term, enforcing consistency constraints on outputs to enhance the performance of individual modalities. For instance, Sarkar et al. <cit.> use KL divergences to align audio and video outputs, while another method, MCKD <cit.>, applies multimodal contrastive KD for video-text retrieval. Although these methods show potential in cross-modal KD, most widely used approaches still rely on single-modal techniques, raising questions about their effectiveness and limitations. This paper, therefore, analyzes key factors influencing cross-modal KD to enhance its application.








 §.§ Domain Gaps in Cross-modal KD


In cross-modal KD, domain gaps critically impact performance. To reduce these gaps, modality alignment methods are often employed to unify different modalities from various domains into a common <cit.> or intermediate modality <cit.>. However, this approach may result in the loss of modality-specific characteristics. To address this, some methods incorporate decoupling strategies to preserve these characteristics, such as using independent detection heads <cit.> for KD or employing feature partitioning to effectively transfer knowledge from the teacher modality while preserving unique features of the student modality <cit.>. Additionally, other approaches focus on minimizing domain gaps by filtering out features with significant discrepancies. For example, Zhuang et al. <cit.> and Wang et al. <cit.> focus on evaluating and filtering significant domain discrepancies, while Huo et al. <cit.> selectively filter out misaligned samples to avoid modality imbalance. Although these methods have made some progress, they primarily address multimodal feature fusion and are not directly applicable to dual-branch networks in cross-modal KD. To address this issue, this paper proposes a mask-based approach to mitigate the impact of domain shifts on cross-modal KD.








§ THE PROPOSED HYPOTHESIS
 
In this section, we first define the symbols and present the necessary assumptions for subsequent proofs. Next, we introduce the Non-target Divergence Hypothesis and provide a detailed explanation of this hypothesis based on experimental results on Scikit-learn dataset <cit.>. Finally, we prove the hypothesis using Vapnik-Chervonenkis (VC) theory <cit.>.



 §.§ Symbol Definitions And Conditional Assumptions
 

We provide a comprehensive overview of the fundamentals of KD and introduce the symbols used in this study. Although our focus is primarily on C-class classification problems, the concepts discussed are also applicable to regression tasks. To maintain the generality of the framework, we consider a two-modal setting, where the data is represented as x^a and x^b, corresponding to the data from modalities `A' and `B', respectively, as shown in Eq. (<ref>).


    {{(x_i^a,y_i)}_i=1^n∼P^n(x^a,y),x_i^a∈ℝ^d_a,y_i∈Δ^c
    
    
       {(x_i^b,y_i)}_i=1^n∼P^n(x^b,y),x_i^b∈ℝ^d_b,y_i∈Δ^c
    .,



where (x_i^a,y_i) and (x_i^b,y_i) respectively represent the feature-label pair of modality `A' and modality `B', and Δ^c represents the set of c-dimensional probability vectors. 

Suppose that our goal is to train a student network that takes x^b as input. In the case of cross-modal KD, the teacher network takes the training data x^a as input and minimizes the training objective as follows:


    f_t=f∈ℱ_tmin 1/n∑_i=1^nℒ(y_i,σ (f(x_i^a)))+Ω ( f ) ,


where ℱ_t is a class of functions from ℝ^d_a to ℝ^c , the function σ : ℝ^c→Δ^c is the softmax operation


    σ(z)_k=e^zk/∑_j=1^ce^z_j.


For all 1≤ k≤ c , the function  ℒ:Δ^c×Δ^c→ℝ is Kullback-Leibler(KL) Divergence <cit.>, and Ω :ℝ→ℝ is an increasing function which serves as a regularizer.


After training the teacher model f_t using the data x_i^a from modality `A', our goal is to transfer the knowledge acquired by the teacher network to the student network operating in modality `B'. Therefore, in addition to minimizing the KL loss between the student output and one-hot label, it is also required to minimize the KL loss between the teacher and student outputs. The objective of optimizing the student network is as follows:



    f_s   = f ∈ℱ_smin 1/n∑_i=1^n[(1-λ) ·ℒ(y_i,σ (f(x_i^b))) 
          + λ·ℒ(s_i,σ (f(x_i^b)))],



where s_i=σ (f_t(x_i^a)/T)∈Δ^c  represents the soft predictions obtained from f_t about the training on modality `A'. The temperature parameter T (T>0)  controls the level of softening or smoothing of the class-probability predictions from f_t and the imitation parameter λ∈ [0,1] determines the balance between imitating the soft predictions s_i and predicting the true hard labels y_i.

Given that the primary focus of this paper is the impact of domain gaps in cross-modal KD, we have listed the following assumptions conditions to control variables and exclude the interference of model capacity and modality strength. These assumptions form the foundational conditions for the discussion in this paper and the basis for the theoretical reasoning in Sec. <ref>.



 
  * Assumption 1: ℱ_s and ℱ_t  have the same model capacity, meaning they have the same ability to fit or learn complexity or accommodate information.


  * Assumption 2: x_i^a  and x_i^b have the same modality strength, meaning that when the same model is trained using x_i^a  and  x_i^b as data separately, the difference in model prediction accuracy is not significant.







 §.§ Non-target Divergence Hypothesis






Under the assumption in Sec. <ref> and based on VC theory <cit.>, the upper and lower bounds of the cross-modal KD approximation error have been derived (see Sec. <ref> for the omitted proof). According to this conclusion, it can be inferred that the divergence in non-target class prediction distributions between the teacher and student networks is a key factor affecting the effectiveness of cross-modal KD. This divergence stems from inherent domain discrepancies between modalities, which hinder effective guidance from the teacher to the student. This leads to  the non-target divergence hypothesis.

Non-target Divergence Hypothesis (NTDH): For cross-modal KD, the performance of KD is determined by the distribution divergence of non-target classes. The smaller this divergence, the better the student network performs.


This hypothesis posits that cross-modal KD benefits from the consistency in the distribution of non-target classes. Furthermore, it explains the observation that, in certain circumstances, the performance of the teacher network is not directly correlated with the performance of the student network.


To intuitively understand our hypothesis, we use the Scikit-learn toolkit to simulate six sets of multimodal data, where γ∈[ 0,1 ] represents the degree of domain discrepancy between modalities, with higher values indicating greater domain differences, as shown in Fig. <ref>. To satisfy Assumptions 1 and 2, both the teacher and student networks use the same architecture and maintain consistency in data intensity (see Sec. <ref> for details). We conduct experiments on these six datasets and record the Jensen-Shannon Divergence between the teacher and student networks in both target and non-target distributions during stable training, denoted as TCJSD and NCJSD, respectively:



    TCJSD =1/2· p_k^talog( p_k^ta/m_k)+1/2· p_k^sblog( p_k^sb/m_k),



    NCJSD = ∑_i=1,i k^C( 1/2·p̂_i^talog( p̂_i^ta/m_i) + 1/2·p̂_i^sblog( p̂_i^sb/m_i) ),



where m_i=1/2·( p̂_i^ta+p̂_i^sb),
m_k=1/2·( p_k^ta+p_k^sb),



    [                                                              p_k^ta        = σ (f_t(x_k^a)) = exp (f_t(x_k^a))/∑_j=1^Cexp (f_t(x_j^a)),;                                                              p_k^sb        = σ (f_s(x_k^b)) = exp (f_s(x_k^b))/∑_j=1^Cexp (f_s(x_j^b)),;                                                             p̂_i^ta = σ (f_t(x̂_i^a)) = exp (f_t(x̂_i^a))/∑_j=i, j k^Cexp (f_t(x_j^a)),;                                                             p̂_i^sb = σ (f_s(x̂_i^b)) = exp (f_s(x̂_i^b))/∑_j=i, j k^Cexp (f_s(x_j^b)). ]
 



The experimental results indicate that as the domain discrepancy between modalities increases (i.e., as γ increases from 0 to 1), the accuracy of the teacher network remains stable, while the performance of the student network declines significantly. Concurrently, there is a significant increase in the distance of the non-target distribution, whereas the distance of the target distribution remains almost unchanged, as illustrated in Fig. <ref>. These findings suggest that in cross-modal KD, the greater the distribution divergence of non-target classes, the lower the effectiveness of KD. In contrast, a smaller discrepancy leads to better distillation outcomes. This is a qualitative observation; further validation of these conclusions will be provided through theoretical derivations in Sec. <ref>.





 §.§ Prove the Non-target Divergence Hypothesis


























Recall our three actors: the student function f_s∈ℱ_s  (trained on x_i^b ), the teacher function f_t∈ℱ_t (trained on x_i^a or x_i^b), and the real target function of interest to both the student and the teacher, f∈ℱ . For simplicity, consider pure distillation, where the imitation parameter is set to λ =1.

According to VC theory <cit.>, the classification error of the classifier, f_s^b can be expressed as:


    R(f_s^b)-R(f)≤ O( | ℱ_s^b|_C/√(n))+ε_sb,


where the O(· )  and ε_sb terms are the estimation and approximation error, respectively. The former refers to the performance gap between a model on training data and its theoretical best performance. The latter refers to the difference between a model output and the true target function. It reflects whether the model representational capacity is sufficiently powerful to accurately approximate the true target function. If the hypothesis space of a model cannot capture the complexity of the target function, the approximation error will be large. Here, R represents the error, | ·|_C denotes a measure of the capacity of some function class, and n represents the number of data points.

Let f_t^a∈ℱ_t^a and f_t^b∈ℱ_t^b be the teacher function trained on  x_i^a and x_i^a , then:


    R(f_t^a)-R(f)≤ O( | ℱ_t^a|_C/n)+ε_ta,



    R(f_t^b)-R(f)≤ O( | ℱ_t^b|_C/n)+ε_tb.


Then, we can transfer the knowledge of the teacher separately from data `A' or `B' to the student. Let f_t^a  serve as the teacher function in cross-modal KD, and f_t^b in Unimodal KD, then:

Cross modal KD


    R(f_s^b)-R(f_t^a)≤ O( | ℱ_s^b|_c/n^α)+ε_m,


where  ε_m is the approximation error of the teacher function class ℱ_s^b with respect to f_t^a∈ℱ_t^a , and 1/2≤α≤ 1.

Unimodal KD

    R(f_s^b)-R(f_t^b)≤ O( | ℱs^b|_C/n^α)+ε_l,


where  ε_l is the approximation error of the teacher function class ℱ_s^b  with respect to  f_t^b∈ℱ_t^b.

Then, if we employ cross-modal KD, combining Eqs. (<ref>) and (<ref>), we can obtain an alternative expression for the student learning the real function f, as follows:

Alternative Expression Through Cross-Modal KD


    R(f_s^b)-R(f)=R(f_s^b)-R(f_t^a)+R(f_t^a)-R(f) 
       ≤ O( | ℱ_s^b|_C/n^α) +ε_m+O( | ℱ_t^a|_C/n)+ε_ta.


Similarly, by employing unimodal KD and combining Eqs. (<ref>) and (<ref>), we can obtain an alternative expression for the student's learning of the real function f, as follows:

Alternative Expression Through Unimodal KD


    R(f_s^b)-R(f)=R(f_s^b)-R(f_t^b)+R(f_t^b)-R(f) 
       ≤ O( | ℱ_s^b|_C/n^α) +ε_l+O( | ℱ_t^b|_C/n)+ε_tb.


Combining Eqs. (<ref>) and (<ref>), it is necessary to satisfy Eq. (<ref>); otherwise, cross-modal KD would not outperform Unimodal KD.


    O( | ℱ_t^a|_C/n)_estimation+ε_m+ε_ta_approximation≤O( | ℱ_t^b|_C/n)_estimation+ε_l+ε_tb_approximation.



Observing Eq. (<ref>), we can see that it consists of two components: estimation and approximation error. Next, we will analyze these two parts separately. 

Regarding estimation error, it is primarily determined by model capacity and data pattern strength. According to the assumptions in Sec. <ref>, when the model capacity and data strength are the same, O(| F_t^a|_C/n) and O(| F_t^b|_C/n) are equivalent. Therefore, we can eliminate O(| F_t^a|_C/n) and O(| F_t^b|_C/n) from both sides of Eq. (<ref>) without altering its outcome. 

Regarding the approximation error term, it reflects the difference between the output of the neural network or the KD model and the target. In a neural network model, the model has only one objective: to minimize the difference between the network's predictions and the one-hot encoded target. Here, ε_ta and ε_tb represent the approximation errors when training the neural network model under the modality a and the modality b, respectively. Under the conditions of Assumption 2, when the intensities of the data modalities are the same, the error between the prediction results trained under modality a and modality b and the ground truth are the same. Therefore, we can further eliminate ε_ta and ε_tb on both sides of Eq. (<ref>). In the KD model, the model has two objectives: first, to minimize the discrepancy between the student output and the one-hot target; second, to reduce the difference between teacher and student outputs. ε_l represents the unimodal KD model, while ε_m represents the multimodal KD model. For the error generated by the first objective, regardless of whether the prediction results come from the unimodal or multimodal model, they are compared with the ground truth. When both Assumption 1 and Assumption 2 are satisfied, the approximation errors of the models is the same and can therefore be offset against each other. For the error generated by the second objective, due to the different targets of the unimodal and multimodal models (the unimodal model aims to approximate the prediction results under modality a, while the multimodal model aims to approximate the prediction results under modality b), they cannot be offset against each other. For the unimodal model, if the training time is sufficient and Assumption 1 is met, the error will approach zero because the input modalities of the teacher and student networks are the same. In contrast, due to the modality differences in the multimodal model, the error is larger and cannot be ignored.


Based on the above analysis, we simplify Eq. (<ref>) by canceling O(| F_t^a|_C/n) with O(| F_t^b|_C/n) and ε_ta with ε_tb on both sides, ignoring ε_l, and retaining ε_m, ultimately reducing Eq. (<ref>) to:


    ε_m<0.


Since ε_m uses the KL divergence for optimization in cross-modal KD, and according to Gibbs' inequality, the KL divergence is nonnegative, ε_m must also satisfy the following:


    ε_m=KL(F_t^a||F_s^b)≥ 0.


By comparing Eq. (<ref>) with Eq. (<ref>), we arrive at a contradictory conclusion, indicating that the condition for cross-modal KD to outperform unimodal KD cannot hold. In other words, when both Assumption 1 and Assumption 2 are satisfied, the performance of cross-modal KD cannot exceed that of unimodal KD. Thus, we obtain the lower bound for the approximation error in cross-modal KD, which corresponds to unimodal KD. In this case, unimodal KD can be regarded as a special case of cross-modal KD, where there are no modality differences.

Next, we will further derive the upper bound of the approximation error ε_m in cross-modal KD. According to <cit.>, KL divergence can be decomposed into a part related to the target class distribution and a part related to the non-target class distribution, denoted by KL(p^ta||p^sb) and KL(p̂^ta||p̂^sb) respectively. The decomposed result is represented as:

    ε_m=KL(F_t^a||F_s^b)=p_k^talog(p_k^ta/p_k^sb)+∑_i=1,i k^Cp_i^talog(p_i^ta/p_i^sb) 
        =p_k^talog(p_k^ta/p_k^sb)+p_\ k^ta∑_i=1,i k^Cp̂_i^ta(log(p̂_i^ta/p̂_i^sb)+log(p_\ k^ta/p_\ k^sb)) 
        =p_k^talog(p_k^ta/p_k^sb)+p_\ k^talog(p_\ k^ta/p_\ k^sb)_KL(p^ta||p^sb)+p_\ k^ta∑_i=1,i k^Cp̂_i^talog(p̂_i^ta/p̂_i^sb)_KL(p̂^ta||p̂^sb)
        =KL(p^ta||p^sb)+p_\ k^ta· KL(p̂^ta||p̂^sb) 
    ,


where 


    [                                                                     p_i^ta               = σ (f_t(x_i^a)) = exp (f_t(x_i^a))/∑_j=1^Cexp (f_t(x_j^a)),;                                                                     p_i^sb               = σ (f_s(x_i^b)) = exp (f_s(x_i^b))/∑_j=1^Cexp (f_s(x_j^b)),;                                                                   p_\ k^ta = σ (f_\ k(x_i^a)) = ∑_i=1, i k^Cexp (f_t(x_i^a))/∑_j=1^Cexp (f_t(x_j^a)).;                                                                            ]


Based on Eq. (<ref>), the upper bound of the approximation error in cross-modal KD can be obtained as follows:



    ε_m=KL(p^ta||p^sb)+p_\ k^ta· KL(p̂^ta||p̂^sb) 
       ≤KL(p^ta||p^sb)_TCKL+KL(p̂^ta||p̂^sb)_NCKL.



According to Eq. (<ref>), the upper bound of the approximation error in cross-modal KD is composed of two parts: one part is determined by the distribution error of the target classes, referred to as TCKL; the other part is determined by the distribution error of the non-target classes, referred to as NCKL. Compared to TCKL, NCKL involves the probability distributions of more classes, thereby increasing its complexity and uncertainty. It can be proven that when the number of non-target classes exceeds two, NCKL becomes significantly larger than TCKL. To demonstrate this, consider the expressions for TCKL and NCKL:




    TCKL=p_k^talog(p_k^ta/p_k^sb) ,



    NCKL=∑_i=1,i k^Cp̂_i^talog(p̂_i^ta/p̂_i^sb).


The ratio of NCKL to TCKL can be derived from Eqs. (<ref>) and (<ref>), as shown in


    NCKL/TCKL=∑_i=1,i k^Cp̂_i^talog(p̂_i^ta/p̂_i^sb)/p_k^talog(p_k^ta/p_k^sb).


To estimate NCKL/TCKL, we perform a Taylor series expansion and approximate by neglecting higher-order infinitesimal terms. Let p̂_i^sb=p̂_i^ta+ε_i and p_k^sb=p_k^ta+ε_t, where ε_i and ε_t are small quantities. After the Taylor series expansion, the result of log( p̂i^ta/p̂_i^sb) is as follows:


    log( p̂_i^ta/p̂_i^sb)≈log( 1-ε_i/p̂_i^ta)≈ -ε_i/p̂i^ta-1/2( ε_i/p̂_i^ta)^2.


Therefore, NCKL and TCKL can be approximated by:


    NCKL   ≈∑_i kp̂_i^ta( -ε_i/p̂_i^ta-1/2( ε_i/p̂_i^ta)^2) 
       = -∑_i kε_i-1/2∑_i tε _i^2/p̂_i^ta,



    TCKL≈ p_k^talog( p_k^ta/p_k^ta+ε_t)≈ -p_k^taε_t/p_k^ta=-ε_t.


From Eqs. (<ref>) and (<ref>), NCKL/TCKL can be approximated by:


    NCKL/TCKL≈-∑_i kε_i-1/2∑_i kε _i^2/p̂_k^ta/-ε_t≥C-1/1·ε_i/ε_t.



Based on Assumptions 1 and 2, the prediction errors of the teacher and student networks on the target class are comparable, so ε_t is much smaller than ε_i. When C-1 is sufficiently large, NCKL/TCKL will be much greater than 1. In summary, we have proven that in cross-modal KD, when the number of classes C is greater than 2, NCKL is significantly larger than TCKL. Therefore, the distribution differences among the non-target classes are the key factors that influence the effectiveness of cross-modal KD. When the distribution differences among non-target classes decrease, the upper bound of the approximation error will correspondingly reduce, thereby enhancing the effectiveness of cross-modal KD.

































































§ EXPERIMENTS






In this section, we validate the hypothesis through experiments conducted on five multimodal datasets. First, we introduce the datasets and network models used in the experiments. Next, we elaborate on the experimental design and methodology. Finally, we assess the validity of the hypothesis in the context of traditional KD <cit.>. Additionally, we demonstrate the broad applicability and practical value of NTDH.



 §.§ Datasets


The five multimodal datasets encompass three types: a simulated dataset (Scikit-learn), a synthetic dataset (MNIST/MNIST-M), and three real-world datasets (RAVDESS, SemanticKITTI, and NYU Depth V2). To ensure that the experimental datasets adhere to the assumptions outlined in Section <ref> and to mitigate the effects of data modality strength and model capacity on the results, we perform preprocessing. The adjustments to model capacity and modality strength are summarized in Table <ref>. 








  §.§.§ Scikit-learn



The Scikit-learn dataset is generated using the Scikit-learn toolkit, which allows us to precisely control the differences and strengths between modalities, ensuring that the data fully meets Assumptions 1 and 2. Specifically, we use the make_classification function from the toolkit to generate multi-class data, where the labels are determined by the feature f, and the length of the feature is d. Next, let x^a∈ℝ^d, x^b∈ℝ^d, and y constitute the multimodal data (x^a,x^b,y). Each modality is formed by a portion of the decisive features of x combined with noise z∈ℝ^d_z. Specifically, x^b=[ x_d_0:d_b,z ]∈ℝ^d, x^a=[ x_d_0:d_t,x_d_b:d_a,z ]∈ℝ^d, and d_a+d_t=2·d_b. We define a ratio γ =d_a-d_b/d_b ∈ [0,1], which characterizes the ratio of domain difference between modalities. As γ increases, the domain difference between data modalities also increases. In the experiment, x^b∈ℝ^d is fixed, and γ is set to γ =[0,0.25,0.5,0.75,1], constructing data with varying modality domain differences, as shown in Fig. <ref>(a).



  §.§.§ MNIST/MNIST-M
 
MNIST <cit.> is a widely used dataset for handwritten digit recognition, containing grayscale images of digits from 0 to 9, each paired with a corresponding label. Each image has a resolution of 28x28 pixels, with the dataset comprising 60,000 training images and 10,000 test images. MNIST-M <cit.> is derived from the MNIST digits by randomly mixing colored patches from the BSDS500 <cit.>, creating a different modality of the same handwritten digits. Since the accuracy performance of MNIST and MNIST-M varies under the same model, to satisfy Assumption 2, which requires that the data strengths of both modalities be the same, we add noise of varying intensities into MNIST and MNIST-M, as shown in Fig. <ref>(b).



  §.§.§ RAVDESS

The RAVDESS <cit.> is a dataset used for multimodal emotion recognition, containing data in both audio and video modalities. The dataset consists of speech and song recordings by 24 actors (12 male and 12 female), covering emotional categories such as neutral, happy, angry, fearful, disgusted, surprised, and sad. RAVDESS includes a total of 1,440 files and is widely used in research on emotion recognition, multi-modal learning, and human-computer interaction systems. In our study, we use the video modality as the input for the student network, while the audio modality serves as the input for the teacher network.



  §.§.§ SemanticKITTI


SemanticKITTI <cit.> is a large-scale dataset based on the KITTI odometry benchmark, providing 43,000 scans with point-wise semantic annotations, of which 21,000 scans (sequences 00-10) are available for training and validation. The dataset includes 19 semantic categories and is used for the evaluation of point cloud semantic segmentation benchmarks. To enable different modalities to be processed by the same network, we follow <cit.> and project the original 3D point cloud data into the camera coordinate system, resulting in 2D point cloud features, as shown on the left side of Fig. <ref>(c).

Additionally, in point cloud semantic segmentation, the image modality lacks dense semantic labels, leading to significantly higher segmentation accuracy for the point cloud modality compared to the image modality. To equalize the data strength, we obtain dense semantic labels for the image modality following the method described in <cit.>, as shown on the right side of Fig. <ref>(c).




  §.§.§ NYU Depth V2
 
The NYU Depth V2 dataset, introduced by Silberman et al. in 2012, is collected using Microsoft Kinect's RGB and depth cameras from commercial and residential buildings across three cities in the United States. The dataset consists of 1,449 densely annotated pairs of RGB and depth images, covering 464 distinct indoor scenes across 26 scene categories. In this study, the RGB data serve as the input for the teacher network, while the depth images are used as input for the student network, and experiments are conducted in the context of the semantic scene completion task.




 §.§ Network Model

We fine-tune the existing network models to serve as the teacher and student models for our experiments. Specifically, for the Scikit-learn, both the teacher and student networks use a 2-layer fully connected network; for the MNIST/MNIST-M, both employ a 3-layer fully connected network; for the RAVDESS, due to the differing shapes of audio and image data, both the teacher and student networks utilize a 2D convolutional network with the same number of layers, and the kernel sizes are adjusted to accommodate the different data modalities. In the SemanticKITTI, the overall network architecture proposed by <cit.> is adopted, with the intermediate fusion layer removed; similarly, in the NYU Depth V2, the network architecture proposed by <cit.> is used, and the fusion layer is also removed.



 §.§ Experiment Plan




To verify the validity of our hypothesis, we design a detailed experimental plan. We employ a divide-and-conquer approach by splitting the hypothesis into two parts and validating each separately. For the first part, we use a weight adjustment method to verify that the distribution differences among non-target classes are the primary factors affecting the effectiveness of KD. This is done by adjusting the weight coefficients in the loss function that account for the distribution differences between target and non-target classes. For the second part, we apply a masking method to remove features or samples that have a significant impact on the distribution differences among non-target classes, observing whether this can improve the performance of cross-modal KD. The experimental plan is illustrated in Fig. <ref>, with the detailed steps as follows:



  §.§.§ The First Part
 We observe whether the distribution differences among non-target classes are the primary factors affecting KD by altering the weights in the loss function that optimize the distribution differences between target and non-target classes. We refer to this method as the weight adjustment method. Specifically, we first decompose the KL divergence-based KD loss function into TCKL and NCKL according to Eqs. (<ref>) and (<ref>). TCKL aims to reduce the distribution gap between the teacher and student for target classes, while NCKL focuses on reducing the distribution gap for non-target classes. Next, we assign weight coefficients α and β to TCKL and NCKL, respectively. Finally, by adjusting these weight coefficients, we either increase or decrease the influence of the distribution divergence of non-target classes.



  §.§.§ The Second Part
 In the first part of the study, we confirm that the distribution differences among non-target classes are a key factor affecting the effectiveness of cross-modal KD. To further explore the relationship between non-target distribution differences and KD performance, we propose a mask-based method. This method ranks the features or samples based on the non-target distribution differences. After ranking, we only use the top-ranked features or samples during the distillation process to reduce the non-target distribution differences. If this method improves the performance of KD, it validates our hypothesis. For smaller datasets, such as the Scikit-learn dataset, MNIST/MNIST-M, and RAVDESS, we apply the feature mask, while for larger datasets, such as SemanticKITTI and NYU Depth V2, we use the sample mask. The specific algorithm is as follows:







Feature Mask: The main steps of the feature mask method are shown in Algorithm 1. The inputs are x^a∈ℝ^n×d_1, x^b∈ℝ^n×d_2, and y∈ℝ^n, representing n pairs of features from modalities a and b, as well as the corresponding labels for these n targets. The output is a saliency vector p∈ℝ^d_1, which represents the difference in non-target class distribution between the teacher and student networks, where the i-th entry p_i∈ [0,1] reflects the saliency of the corresponding feature dimension. A larger saliency value indicates a greater difference in non-target class distribution for that feature channel. 

Algorithm 1 is designed based on a backtracking approach starting from the output layer. Specifically, in Step 4, we jointly train two unimodal networks, f_a and f_b, which take unimodal data x_a and x_b as inputs, respectively. The first term in Eq. (<ref>) aims to align the feature spaces learned by the two networks, while the remaining terms ensure that the learned features accurately predict the labels. In Step 5, we utilize the idea of feature importance ranking <cit.> to trace the saliency of input features  with respect to non-target class distribution differences. For the i-th dimension in x_a, we randomly permute the values along that dimension and obtain the permuted result x̃_a in Step 8. Next, in Step 10, we quantify the difference in non-target class distribution for each input feature channel by calculating the NCJSD, where a larger distance indicates a more significant difference between the teacher and student networks. Therefore, we can quantify the magnitude of the non-target class distribution difference in each input feature channel using the saliency vector p.

We repeat the permutation process multiple times and average the distance values to improve stability. Finally, in Step 13, p is normalized to [0,1]. Once the feature channels are ranked, we can reduce the non-target class distribution differences during distillation by applying a mask to the top γ% of the feature channels.





Sample Mask: The main steps of the feature masking method are illustrated in Algorithm 2. The inputs to Algorithm 2 are σ (f_t(x_i^a)) and σ (f_s(x_i^b)), representing the predicted probability distributions of the teacher and student networks, respectively. The output is a saliency vector P∈ℝ^n, which indicates the difference in non-target class distribution between the teacher and student networks, where the i-th entry p_i∈ [0,1] reflects the saliency of the corresponding sample. A larger saliency value indicates that the sample exhibits a greater difference in non-target class distribution. Here, n represents the number of samples, such as the number of points in each batch for the SemanticKITTI or the number of pixels in each batch for the NYU Depth V2.

In Step 6, we calculate the NCJSD for all samples in each batch. In Step 8, we sort these distances, where larger distances indicate more significant differences in non-target class distribution between the teacher and student networks. Therefore, we can quantify the magnitude of the non-target class distribution difference for each sample using the saliency vector p. Finally, in Step 12, p is normalized to [0,1]. Once the sorting of all samples in the batch is completed, we can reduce the distribution differences between the non-target classes during distillation by applying a mask to the top γ% of the samples.










 §.§ The Results of the First Part



  §.§.§ Scikit-learn






The results of the Scikit-learn dataset are shown in Fig. <ref>. The horizontal axis represents the modality difference γ, where γ = 0 indicates no difference and γ = 1 indicates the maximum difference. The experiment sets six levels of modality divergence. By adjusting the weights α and β, we study the impact of distribution divergence between target and non-target classes on the effectiveness of KD.

In Fig. <ref>(a), we fix α and increase β to give more weight to the target class under the six different modality differences. Correspondingly, in Fig. <ref>(b), we fix the β and increase the α to give more weight to the distribution divergence of non-target classes. By comparing the experimental results, it can be observed that when γ =1, indicating a large modality difference, increasing α significantly improves the effectiveness of KD, whereas increasing β significantly reduces the effectiveness. This indicates that, in cases of large modality differences, the distribution divergence of non-target classes play a crucial role in the effectiveness of KD.






  §.§.§ Other dataset




On the MNIST/MNIST-M datasets, we employ the feature mask, fixing the β parameter while gradually increasing the α to amplify the weight of the distribution divergence of non-target classes. We then fix the α and increase the β to enhance the weight of target class distribution differences. The results show that when the weight of non-target class distribution divergence increases, accuracy significantly decreases, while increasing the weight of target class distribution divergence results in almost no noticeable change. This indicates that the distribution divergence of non-target classes is a crucial factor influencing the effectiveness of cross-modal KD. To further validate this phenomenon, similar  experiments are conducted on the RAVDESS, SemanticKITTI, and NYU Depth V2 datasets, and consistent results are obtained, as shown in Figs. <ref>(b)-(d), respectively. These consistent results across different datasets further confirm our hypothesis.




 §.§ The Results of the Second Part



  §.§.§ Scikit-learn




By applying Algorithm 1 to deactivate the top γ of features with the highest NCJSD, we reduce the non-target distribution divergence between the teacher and student networks, which is referred to as the True Mask. For control purposes, we establish two additional groups: one that deactivates the top γ of features with the lowest NCJSD, thereby increasing the non-target distribution divergence between the teacher and student networks, is known as the False Mask; the other randomly deactivates γ of features, referred to as the Random Mask.



Fig. <ref> shows the results for γ = 0.5 and γ = 0 on the Scikit-learn dataset. The 0-th layer features of the teacher network (i.e., the input data) are selected as the masking object, with nine different masking ratios ranging from 10% to 90%. For the True Mask, a higher removal ratio indicates that more features with greater non-target distribution divergence are removed from the teacher network. Conversely, for the False Mask, the situation is reversed. The results show that for the True Mask, the performance of the student network initially improves with increasing γ%. This improvement suggests that features with significant non-target distribution divergence between the teacher and student networks are being progressively discarded, thereby enhancing the performance of the student network. However, as the deactivation process starts to affect features with smaller non-target distribution divergence, performance begins to decline. In contrast, the False Mask performs the worst, while the Random Mask results fall between the two. This observation is consistent with our expectations.




  §.§.§ MNIST/MNIST-M and RAVDESS




Similarly, we also conduct experiments on the MNIST/MNIST-M and RAVDESS datasets using Algorithm 1, and reach similar conclusions, as shown in Figs. <ref>(a) and (b). Additionally, we use heatmaps to visualize the masked areas on the MNIST/MNIST-M datasets, as shown in Fig. <ref>. For the False Mask, the highlighted regions are mainly concentrated in the non-target areas, with fewer in the target areas; for the True Mask, the situation is the opposite, with the highlighted regions primarily focused on the target areas and fewer in the non-target areas. From these results, it is observed that the target areas are mainly concentrated in the foreground, while the non-target areas are concentrated in the background, which aligns with our previous analysis.







  §.§.§ SemanticKITTI and NYU Depth V2




We use Algorithm 2 on the SemanticKITTI and NYU Depth V2 datasets, as shown in Figs. <ref>(a) and (b). The results indicate that the effectiveness of KD initially improves and then declines as samples with large non-target class distribution divergence are removed. Initially, removing these samples enhances KD because their significant distribution differences hinder the process. However, as the removal proportion γ% increases, samples with smaller distribution divergences are also eliminated, leading to a decrease in KD effectiveness in later stages.

Furthermore, we visualize the samples with significant distribution divergence in non-target classes on the SemanticKITTI dataset, as shown in Fig. <ref>. The highlighted areas, primarily concentrated along object edges, indicate that distribution divergence is greatest in these regions. The reason for this is that in segmentation tasks, the edges are typically the most challenging to distinguish, leading to the most pronounced divergence in non-target class distributions between different modalities.







 §.§ Generalization


Beyond traditional KD <cit.>, various improved algorithms have been proposed. To evaluate the generalization ability of NTDH, we apply the feature masking method to several of these improved KD algorithms, including FitNet <cit.>, Contrastive Representation Distillation (CRD) <cit.>, Relational Knowledge Distillation (RKD) <cit.>, Probabilistic Knowledge Transfer for deep representation learning (PKT) <cit.> , Similarity-Preserving KD (SP) <cit.> and Decoupled Knowledge Distillation (DKD) <cit.>.


Table <ref> presents the experimental results on the MNIST/MNIST-M dataset. The results show that none of the methods improve student performance, indicating that KD methods designed for unimodal scenarios are not suitable for cross-modal applications. However, when we use Algorithm 1 to remove portions with significant non-target class distribution divergence, especially at smaller removal proportions (e.g., 25% and 50%), all algorithms show significant performance improvements. In contrast, random and false masking lead to substantial performance declines, highlighting the critical role of non-target class differences in cross-modal KD.

Table <ref> presents the experimental results on the RAVDESS dataset. The results show that cross-modal KD fails to improve student performance, highlighting the limitations of these methods in cross-modal applications. However, using Algorithm 1 to remove portions with significant modal differences, particularly at lower removal proportions (e.g., 25% and 50%), leads to performance improvements in most algorithms. In contrast, control experiments with Random and False masking result in significant performance declines, underscoring the decisive role of non-target class distribution divergence in cross-modal KD.













 §.§ Applications


This section further explores the practical application of NTDH. According to this hypothesis, in cross-modal KD, if the non-target class distribution difference between Teacher (a) and the student is smaller than that between Teacher (b) and the student, then we expect the student guided by Teacher (a) to outperform the student guided by Teacher (b). To reduce the non-target class distribution difference between the teacher and student networks, the method in Algorithm 2 can be applied. To validate the effectiveness of this method, we apply it to existing cross-modal KD algorithms, such as 2DPASS <cit.> and PMF <cit.>, and conduct tests on the SemanticKITTI dataset.

Table <ref> shows the experimental results, indicating that appropriately masking samples with significant non-target distribution divergence (e.g., 25% and 50%) can improve performance. For 2DPASS, the maximum improvement reached 2.0%. By comparing the change curves of NCJSD and TCJSD distances before and after masking, as shown in Fig. <ref>(a), it is evident that the non-target distribution divergence significantly decreases, while the target distribution divergence changes only slightly. This confirms that the performance improvement is due to the reduction of non-target distribution differences. In contrast, the improvement in PMF is smaller because the Perception-Aware Loss in the original paper has already removed samples with large modality differences, leading to insignificant changes in NCJSD and TCJSD distances before and after masking, as shown in Fig. <ref>(b).















§ CONCLUSION

In this work, we delve into cross-modal KD and introduce the NTDH method, which analyzes the impact of domain gaps in multimodal data on KD performance, highlighting the importance of non-target class distribution divergence. Through theoretical analysis and carefully designed experiments, we validate the rationale, generalization capability, and potential applications of NTDH. We aim for NTDH to provide valuable insights for the practical use of cross-modal KD and to foster interest in understanding domain discrepancies in multimodal learning. However, as this paper focuses on the effects of domain discrepancies on cross-modal KD, the exploration of theoretical applications is limited.

































IEEEtran



































































