Unsupervised Domain Adaptation via Calibrating Uncertainties
Ligong Han Yang Zou Ruijiang Gao Lezi Wang Dimitris Metaxas
cs.LG cs.CV stat.ML

































1]Ligong Han
2]Yang Zou
3]Ruijiang Gao
1]Lezi Wang
1]Dimitris Metaxas
[1]Department of Computer Science, Rutgers University
[2]Department of Electrical and Computer Engineering, Carnegie Mellon University
[3]McCombs School of Business, The University of Texas at Austin
[ ]              





Unsupervised Domain Adaptation via Calibrating Uncertainties
    [
    December 30, 2023
============================================================




    Unsupervised domain adaptation (UDA) aims at inferring class labels for unlabeled target domain given a related labeled source dataset. Intuitively, a model trained on source domain normally produces higher uncertainties for unseen data. In this work, we build on this assumption and propose to adapt from source to target domain via calibrating their predictive uncertainties. The uncertainty is quantified as the Rényi entropy, from which we propose a general Rényi entropy regularization (RER) framework. We further employ variational Bayes learning for reliable uncertainty estimation. In addition, calibrating the sample variance of network parameters serves as a plug-in regularizer for training[This is an extended version of a CVPRW'19 workshop paper with the same title. In the current version the gradient variance regularizer is discussed as a new quantification of uncertainty and a way of regularizing the training dynamics.]. We discuss the theoretical properties of the proposed method and demonstrate its effectiveness on three domain-adaptation tasks.





§ INTRODUCTION

The ability to model uncertainty is important in unsupervised domain adaptation (UDA). For example, self-training <cit.> often requires the model to reliably estimate the uncertainty of its prediction on target domain in the pseudo-label selection phase. However, traditional deep neural networks (DNN) can easily assign high confidence to a wrong prediction <cit.>, thus are not able to reliably and quantitatively render the uncertainty given data.

On the one hand, Bayesian neural networks (BNN) <cit.> tackles this problem by taking a Bayesian view of the model training. Instead of obtaining a point estimate of weights, BNN tries to model the distribution over weights. We leverage BNN as a powerful tool for uncertainty estimation. On the other hand, one can estimate the empirical uncertainty of the model by the variance of network parameters, which we call gradient variance regularization (GVR).

Finally, our approach builds on the intuition that a model gives similar uncertainty estimates on the two domains learns to adapt from source to target well. Thus, we propose to directly calibrate the estimated uncertainties between source and target domains during training. This calibration can be considered in three-folds, from which we listed our contributions as follows:
nolistsep

    
  * We propose a general framework for unsupervised domain adaptation by calibrating the predictive uncertainty, and discuss its relationship with entropy regularization <cit.> and self-training <cit.>.
    
  * We introduce variational Bayes neural networks to provide reliable uncertainty estimations.
    
  * We propose to calibrate the variance of network parameters as a model-and-objective-agnostic regularization (GVR) on the optimization dynamics.
    




§ RELATED WORK

Shannon entropy is commonly used to quantify the uncertainty of a given distribution. Entropy-based UDA has already been proposed in <cit.>. Unlike <cit.>, we avoid using adversarial learning which tends to be unstable and hard to train. Also, entropy regularization is proposed in  <cit.> for semi-supervised learning and can be directly applied to UDA. However, our framework is more general since the uncertainty is not necessarily to be the Shannon entropy. In fact, we formalize the uncertainty as Rényi entropy which is a generalization of Shannon entropy. Many other methods in UDA can be modeled under this framework, for example, self-train <cit.> can be viewed as minimizing the min-entropy which is a special case of Rényi entropy.

As pointed out by <cit.>, directly optimizing the estimated Shannon entropy given data requires the classifier to be locally-Lipschitz <cit.>. Co-DA <cit.> and DIRT-T <cit.> propose to solve this problem by incorporate the locally-Lipschitz constraint via virtual adversarial training (VAT) <cit.>. 

Another complimentary line of research employs self-ensemble and shows promising results <cit.>. Indeed, BNN <cit.> performs Bayesian ensembling by nature. This is part of the reason why BNN provides a better uncertainty estimation.


In supervised learning, regularization is proposed to avoid overfitting. Besides weight decay, typical regularization techniques include label smoothing <cit.>, network output regularization <cit.>, knowledge distillation <cit.>. We believe our proposed gradient variance regularizer GVR can also be used in supervised settings.



§ UNCERTAINTY IN DEEP NEURAL NETWORKS

Rényi entropy. For a discrete probability distribution P=(P_1, …, P_K), the Rényi entropy <cit.> of order α (α > 0) is defined as

    H_α(P)=1/1-αlog(∑_kP_k^α).

The limiting value of H_α when α→ 1 is the Shannon entropy, and α→∞ corresponds to the min-entropy, H_∞(P)=min_k-log(P_k)=-logmax_k P_k. A typical deep neural network for classification usually produces a discrete distribution over possible classes given the input data. Thus, we quantify the predictive uncertainty by the Rényi entropy on this probability distribution.

Bayesian neural networks. BNN estimates the posterior over network weights while optimizing the training objective. Given the dataset 𝒟={x^(i), y^(i)}_i=1^N, the output of BNN is denoted as f(x|w) where x is input data and w are the weights. For a classification task, f is the predicted logits and the resulting probability vector is given by a softmax function: P(y|x,w)=softmax(f(x|w)). The predictive distribution over labels given input x is P(y|x)=𝔼_P(w|𝒟)P(y|x,w). We define the uncertainty evaluated by BNNs as the entropy H_α(P(y|x)).

We adopt the method from <cit.>, where aleatoric and epistemic uncertainties are jointly modeled. In <cit.>, the logits are assumed to be Gaussian and the reparameterization trick is utilized. The predicted logit is f̂(x) = μ_θ(x) + σ_θ(x) ϵ with ϵ∼ N(0, I). The final predicted probability vector P(y|x) is approximated by Monte Carlo sampling (with M samples),

    P(y|x; θ) = 1/M∑_m=1^Msoftmax(f̂^(m)(x)).


Variational inference. As estimating the posterior P(w|𝒟) is often intractable <cit.>, variational inference is commonly adopted, where the posterior of weights is approximated by Q_θ(w)≈ P(w|𝒟) with parameter θ. Specifically, in supervised learning, Q_θ(w) is estimated by maximizing the evidence lower bound (ELBO) <cit.>:

    ELBO = 𝔼_Q_θ(w)logP(y|x,w)_(I)-D_KL(Q_θ‖ P(w))_(II),

where P(w) is the prior, and term (I) is the standard cross-entropy loss evaluated at x with parameter w. Gal  <cit.> proposes to view dropout together with weight decay as a Bayesian approximation, where sampling from Q_θ is equivalent to performing dropout and term (II) becomes L_2 regularization (or weight decay) on θ.


Gradient variance. Rather than finding a variational approximation of the posterior Q_θ(w), one can instead estimate the model-dependent uncertainty by the sample variance of θ (or the sample variance of w in the case of non-Bayesian networks). To be precise, sampling mini-batches ℬ_i from a batch ℬ, one can compute the adapted parameters by performing one gradient step (at θ): θ'_i=θ-η∇_θℒ_ℬ_i(f_θ), where ℒ is the objective and η is the inner learning-rate. Then the variance of θ can be defined as the trace of the covariance of vectorized θ'_is: 

    Var({θ'_i})=trace(Cov({vec(θ'_i)})),

where Cov({vec(θ'_i)})=η^2 Cov({vec(∇_θℒ_ℬ_i(f_θ))}) and {·} denotes a collection or a set. It can be easily seen that regularizing the variance of parameters is essentially regularizing the variance of gradients. We will discuss the usage of this gradient variance as a regularizer as well as its relationship with MAML <cit.> in the next section.




§ DOMAIN ADAPTATION VIA CALIBRATING UNCERTAINTIES

Rényi entropy regularization. Denote source and target dataset as 𝒟_S={x^(s), y^(s)}_s∈𝒮 and 𝒟_T={x^(t)}_t∈𝒯 respectively, where x^s,x^t indicate the samples and y^s is the label in source domain, and 𝒟=𝒟_S∪𝒟_T. We propose to calibrate the predictive uncertainty of target dataset with the source domain uncertainties. Concretely, we minimize the cross-entropy (CE) loss in the source domain while constraining the predicted entropy in the target domain:

    min_θℒ_CE   =1/|𝒮|∑_s∈𝒮H_CE(y^(s), P(y|x^(s); θ))
    s.t.     1/|𝒯|∑_t∈𝒯H_α(P(y|x^(t); θ))≤ C,

where H_CE(·,·) is the cross-entropy and C indicates the strength of the applied constraint. In practice, the network is first pretrained on labeled source dataset using ELBO in Equation <ref>. Then, unlabeled target data is introduced in the above Equation <ref>, and P(y|x;θ) is computed from Equation <ref>. Note that the resulting CE loss is no longer the term (I) in ELBO, since the expectation is inside logarithm. We simply treat Q_θ(w) “as is” the true posterior and evaluate CE using P(y|x;θ). For a non-Bayesian network, P(y|x,w) is used as a replacement of P(y|x;θ).

To solve Equation <ref>, rewrite it as a Lagrangian with a multiplier β,

    ℱ=   1/|𝒮|∑_s∈𝒮H_CE(y^(s), P(y|x^(s); θ)) + 
       β( 1/|𝒯|∑_t∈𝒯H_α(P(y|x^(t); θ))-C ).

Since β, C ≥ 0, an upper bound on ℱ is obtained,

    ℱ≤   1/|𝒮|∑_s∈𝒮H_CE(y^(s), P(y|x^(s); θ)) + 
       β/|𝒯|∑_t∈𝒯H_α(P(y|x^(t); θ)) = ℒ_α.

Ideally, Equation <ref> can be optimized via dual gradient descent and β is jointly updated along with θ. For simplicity, we follow the work of <cit.> and fix β as a hyper-parameter in the experiment and minimize the upper bound ℒ_α. 

Note that letting α→ 1 in Equation <ref> is in fact the (Shannon) entropy regularization as described in <cit.>, except that here we consider a variational BNN. As pointed out in <cit.>, directly optimizing Equation <ref> can be difficult and expectation maximization (EM) algorithms are often used. Proposed in <cit.>, deterministic annealing EM anneals the predicted probabilities as soft-labels and minimizes the resulting cross-entropy. In an extreme case, soft-labels become one-hot vectors and the algorithm terms out to be self-training with pseudo-labels <cit.>. In our Rényi entropy regularization framework, self-training is essentially optimizing the min-entropy (α→∞). Then the objective reads

    ℒ_∞=   1/|𝒮|∑_s∈𝒮H_CE(y^(s), P(y|x^(s); θ)) + 
       β/|𝒯|∑_t∈𝒯H_CE(ŷ^(t), P(y|x^(t); θ)),

with ŷ^(t)=onehot(_k∈{1,…,K}P(y_k|x^(t);θ)) to be pseudo-labels in target domain. Subscript k denotes the k-th element in a given K-dim vector. The relationship between ℒ_1 and ℒ_∞ can be immediately realized by noticing that the Shannon entropy is an upper bound of the min-entropy:

    H_1(P)=   -∑_kP_k log(P_k)≥ -∑_kP_klog(max_kP_k)= 
       -log(max_kP_k) =H_∞(P) =H_CE(ŷ, P).


We build our method on top of class-balanced self-training (CBST) proposed in <cit.> and use it as the backbone of RER. CBST seeks to select most confident predictions pseudo-labels in a self-paced (“easy-to-hard”) scheme, since jointly learning the model and optimizing pseudo-labels on all unlabeled data is naturally difficult. The authors also propose to normalize the class-wise confidence levels in pseudo-label generation to balance the class distribution. For a detailed formulation, we suggest readers referring Section 4.1 and 4.2 in <cit.>.

Gradient variance regularization. The entropy regularization or self-training framework as formulated above implicitly encourages cross-domain feature alignment. However, pseudo-labels can be quite noisy even if BNN is employed to estimated their reliability. Trusting all selected pseudo-labels as one-hot-encoded “ground-truth” is overconfident and self-training with noisy pseudo-labels can lead to incorrect entropy minimization. Indeed, we observe that the model can quickly converge to its overconfident predictions. Therefore, the parameter variance evaluated in target domain using pseudo-labels via Equation <ref> can be even smaller than that of the source domain. To address this problem, we again propose to regularize the self-training by maximizing the gradient variance. Algorithm <ref> illustrates the regularized self-training procedure on target domain (the training on source and target domains are preformed alternately, which is omitted in the algorithm box). η and η' are the inner- and outer-stepsize, and λ is the hyper-parameter weighting the regularization term.


Notice that the proposed GVR shares similarities with MAML <cit.>, comparing from a dynamical systems standpoint and despite that MAML samples mini-batches of different tasks. Taking a first-order Taylor expansion of the loss function around θ,

    ℒ_ℬ_i(f_θ'_i) ≈ℒ_ℬ_i(f_θ)-η‖∇_θℒ_ℬ_i(f_θ)‖_2^2,

we demonstrate that MAML tries to maximize the sensitivity of the loss functions with respect to the parameters by maximizing the L_2 norms of the gradients. On the contrary, GVR maximizes the variance of gradients, which intuitively encourages the model to escape from bad local minima.

It is worth mentioning that GVR is not only model-agnostic but also objective-agnostic. This is useful when the regularizer itself is the objective to be optimized. Moreover, GVR is complementary to VAT <cit.> since in VAT the gradient is computed with respect to input data. We conjecture that the data gradient somewhat captures the aleatoric (data-dependent) uncertainty, which we leave for future work.




§ EXPERIMENTS

We first show results on three digit datasets MNIST <cit.>, USPS and SVHN <cit.>, where we consider MNIST→USPS and SVHN→MNIST. Then we present preliminary results on a challenging benchmark: VisDA17 (classification) <cit.> which contains 12 classes. We follow the standard protocol in <cit.>. Classification accuracies on source and target domains for base models are reported in Table <ref>. We use DTN <cit.> as our base model for MNIST→USPS and SVHN→MNIST. To implement its Bayesian variant (BDTN), we add another classifier to predict the logarithm of variance. 





































Domain adaptation results on digit datasets are shown in Table <ref>. Our proposed Rényi entropy regularization methods with non-Bayesian and Bayesian base models are listed as RERs and BRERs respectively. We see self-training with pseudo-labels ((B)RER-∞) are in general more stable than directly minimizing the Shannon entropy ((B)RER-1). Also, adding GVR in (B)RER-∞ improves the performance. However, we also observe that GVR is not helpful in (B)RER-1 settings.



Mean accuracies on VisDA17 dataset are reported in Table <ref>. Following the protocol in <cit.>, we train a standard ResNet101 <cit.> as the base model and add a second classifier (denoted as BRes101) to predict the logarithm of variance on logits. Results show that BNN improves upon non-Bayesian baselines by a large margin. GVR has not been tested on VisDA17 with (B)Res101 since the memory requirement exceeds our GPU capacities.





§ CONCLUSION

In this work, we propose to approach unsupervised domain adaptation via calibrating the predictive uncertainties between source and target domains. The uncertainty is quantified under a general Rényi entropy regularization framework, within which we introduce Bayesian neural networks for accurate and reliable uncertainty estimations. From a frequentist point of view, we in addition propose to approximate the model uncertainty via the sample variance of network parameters (or gradients) during training. Results show that the uncertainty estimation by Bayesian networks and gradient variances is effective and leads to stable performance in unsupervised domain adaptation.


ieee





