# Taxonomy Adaptive Cross-Domain Adaptation in Medical Imaging via
Optimization Trajectory Distillation

Jianan Fan  University of Sydney  Dongnan Liu  University of Sydney  Hang
Chang  Lawrence Berkeley National Laboratory  Heng Huang  University of
Maryland at College Park  Mei Chen  Microsoft  Weidong Cai  University of
Sydney

###### Abstract

The success of automated medical image analysis depends on large-scale and
expert-annotated training sets. Unsupervised domain adaptation (UDA) has been
raised as a promising approach to alleviate the burden of labeled data
collection. However, they generally operate under the closed-set adaptation
setting assuming an identical label set between the source and target domains,
which is over-restrictive in clinical practice where new classes commonly
exist across datasets due to taxonomic inconsistency. While several methods
have been presented to tackle both domain shifts and incoherent label sets,
none of them take into account the common characteristics of the two issues
and consider the learning dynamics along network training. In this work, we
propose optimization trajectory distillation, a unified approach to address
the two technical challenges from a new perspective. It exploits the low-rank
nature of gradient space and devises a dual-stream distillation algorithm to
regularize the learning dynamics of insufficiently annotated domain and
classes with the external guidance obtained from reliable sources. Our
approach resolves the issue of inadequate navigation along network
optimization, which is the major obstacle in the taxonomy adaptive cross-
domain adaptation scenario. We evaluate the proposed method extensively on
several tasks towards various endpoints with clinical and open-world
significance. The results demonstrate its effectiveness and improvements over
previous methods. Code is available at https://github.com/camwew/TADA-MI.

##  1 Introduction

Figure 1: (a) Illustration of the issue of taxonomic inconsistency with nuclei
recognition. The category label sets are incoherent across datasets. Different
colours indicate the class each nucleus belongs to. The red asterisks denote
novel classes only existing in Dataset 2. (b) Concept of the proposed method.
We perform optimization trajectory distillation to deliver external navigation
for learning target domain and new classes where the optimization steps tend
to be restricted and unreliable.

Automated and objective analysis of medical images is an important research
topic and has been explored in various clinical applications [65, 15]. To
relieve the burden of acquiring massive annotated data for model development
on new domains, several unsupervised domain adaptation (UDA) methods [7, 43,
42] are proposed to mitigate the data distribution bias [57] between a richly
labeled source domain and a target domain with no explicit supervision.

However, these works assume a closed-set adaptation setting that the source
and target domains should necessarily share the identical category label space
and definition [49]. The restriction limits their practical applicability in
_the clinical wild_ [17]. Unlike natural images where the definitions of
different entity categories (_e.g._ , cat v.s. dog) are structured and
globally unified, in the medical domain, inconsistency in taxonomy across
different countries or institutes is a common issue compromising the
feasibility of cross-domain adaptation [14]. Take nuclei recognition in
histology images as an example, the ambiguous biological characteristics of
cells and distinct clinical usages of the analysis outcomes result in a lack
of a unified categorization schema [17, 22]. With specific clinical purposes
and down-stream applications, different datasets could categorize nuclei with
disparate granularities of biological concepts. Besides, as medical research
evolves rapidly, novel cell and disease classes could be discovered and form
datasets with continually expanding category label sets [46]. This motivates
us to develop a generalized adaptation method with the capability to tackle
both data distribution bias and category gap for more flexible UDA in clinical
practice.

To this end, we study the problem of taxonomy adaptive domain adaptation [21].
It assumes the target domain could adopt a label space different from the
source domain. Following UDA, a labeled source dataset and unlabeled samples
from the target domain are available during training. Additionally, to
recognize the novel/fine-grained categories which are non-existent or
unspecified in the source domain, a few samples of those target-private
categories are annotated and utilized as exemplars. The technical challenge
for this setting lies in how to alleviate domain shifts and concurrently learn
new classes with very limited supervision. Recently, several methods are
proposed towards a similar goal [35, 24]. However, they typically address the
two issues individually in separate contexts and fail to design a unified
paradigm according to their common characteristics [21], which could incur a
subtle collision across issues since their objectives are non-relevant [68].
Besides, existing works either focus on cross-domain/class alignment in the
feature space [45, 40] or resort to model output-based regularization such as
self-supervision [50], whereas those approaches suffer from the equilibrium
challenge of adversarial learning [1] and the low-quality pseudo-labels [62].

In this work, we present a unified framework for taxonomy adaptive cross-
domain adaptation from a new perspective, _i.e._ , via optimization trajectory
distillation. Our strategy is motivated by a common challenge existing in both
cross-domain and small-data learning regimes, which is the inadequate
navigation in the course of network optimization. For cross-domain learning,
the unstable feature alignment procedure and noisy pseudo-labels tend to
induce error accumulation along network training [71]. Similarly, with limited
support samples in new class learning, the optimization of network is inclined
to step towards restricted local minima and cannot cover a globally-
generalizable space [63]. To this end, we propose to exploit the optimization
trajectory from a reliable “teacher” to provide external guidance for
regularizing the learning dynamics of insufficiently annotated domain and
classes, as illustrated in Fig. 1(b).

Our method consists of two key components, _i.e.,_ cross-domain/class
distillation and historical self-distillation. (i) Cross-domain and cross-
class distillation aim to leverage the optimization trajectory from the richly
labeled domain and classes to serve as the “teacher”. Motivated by the Neural
Tangent Kernel (NTK) theory [29], we characterize the network optimization
trajectory through gradient statistics. Then, given the observation that the
subspace spanned by the gradients from most iterations is generally low-rank
[39, 2], we design a gradient projection approach to suppress the noisy
signals in stochastic gradients and rectify the distillation process.
Thereafter constraints are imposed on the gradient statistics of target domain
and new classes to calibrate their training dynamics towards domain-invariant
and unbiased learning procedure. (ii) Historical self-distillation further
drives the optimization paths of model to converge towards flat minima. It is
found that the flatness of loss landscapes is strongly related to the model’s
robustness and generalizability [28], while how to take advantage of the
insight to tackle domain shifts and limited supervision is under-explored. We
propose to exploit the historical gradients to construct the informative low-
rank subspaces and then perform gradient projection to alleviate loss
sharpness. It compensates for the intense and out-of-order optimization
updates incurred by inadequate regularization and leads to better
generalization.

Our prime contributions are as follows: (1) We introduce a more generalized
cross-domain adaptation paradigm for medical image analysis in which both data
distribution bias and category gap exist across the source and target domains.
(2) We leverage insights from recent learning theory research and propose a
novel dual-stream optimization trajectory distillation method to provide
external navigation in network training. We perform theoretical justifications
from two perspectives to illustrate the merits of our method. (3) Experiments
on various benchmarks validate the effectiveness and robustness of our
proposed method and its improvements over existing approaches.

##  2 Related Works

Domain Adaptation Domain adaptation (DA) aims to mitigate the data
distribution bias [18]. The classic semi-supervised/unsupervised DA focuses on
the scenario where the target domain has an identical label space to the
source domain [53, 16, 36]. The over-restricted precondition cannot suffice
the demand of clinical usages [66]. There exist several efforts to take a step
further than the closed-set adaptation, such as open-set DA [49] and universal
DA [67]. However, the aim of those works is solely to detect the previously
unseen classes, instead of learning to separately identify each new class with
few supports. Some recent studies suggest to explicitly recognize the target-
private categories. [35] performs feature projection and alignment based on
the prototypical networks [56]. [21] combines pseudo-labeling and contrastive
learning to combat domain bias and label gap. Nevertheless, those works are
limited to tackling the two issues individually, which fails to exploit their
common characteristics and propose a unified solution.

Cross-Domain Few-Shot Learning The technical challenge we need to resolve is
similar to an emerging research topic, _i.e.,_ cross-domain few-shot learning
[24]. The mainstream methods for this challenge include self-training [50, 27,
69], contrastive learning [13, 21], feature alignment/augmentation [60, 8,
26], and channel transformation [38, 44]. Those methods either focus on
feature-level modulation or turn to output-level self-supervision and fine-
tuning, which could bring instability along optimization [1, 62]. In this
work, we propose a novel gradient-based framework to transfer knowledge in
both cross-domain and few-shot settings. It implicitly characterizes the
information in both feature space and output space.

Gradient Manipulation In current deep learning systems where gradient descent
serves as the most popular training algorithm, the directions and amplitudes
of network optimization steps are embedded in the gradient space [9]. Gradient
manipulation refers to directly modulating the optimization gradients for
improved learning process [47]. Recently, it is introduced to mitigate the
data distribution shift and shows promises in UDA and domain generalization
[16, 19, 55]. However, those works are still restricted by the closed-set
condition and do not consider the low-rank nature of gradients [39], which
make their approaches vulnerable to noises in the small-data learning regime.

Flatness in Optimization In previous machine learning research, the connection
between convergence to flat minima in optimization and model generalizability
has long been established [25, 34]. Theoretical and empirical studies prove
that the flatness of loss landscapes highly correlates with generalization
capability [4]. [5] averages the network weights of multiple checkpoints to
prompt generalization across domains. Different from their post-hoc averaging
approach operating on fixed models, we propose to regularize the intermediate
dynamics of network optimization to achieve flat minima.

##  3 Method

###  3.1 Problem Statement

The taxonomy adaptive cross-domain adaptation problem can be formalized as
follows. There exists a labeled source dataset
$\mathcal{D}_{s}=\\{(\mathbf{x}_{s},\mathbf{y}_{s})\\}$, an unlabeled target
dataset $\mathcal{D}_{t}^{u}=\\{(\mathbf{x}_{t}^{u})\\}$, and a few-shot
labeled support set from the target domain
$\mathcal{D}_{t}^{l}=\\{(\mathbf{x}_{t}^{l},\mathbf{y}_{t}^{l})\\}$. Samples
from source and target datasets are drawn from different data distributions,
_i.e._ , $\mathbf{x}_{s}\sim\mathcal{P}_{s}$,
$\mathbf{x}_{t}\sim\mathcal{P}_{t}$, $\mathcal{P}_{s}\neq\mathcal{P}_{t}$. The
label sets of source and target datasets are denoted as $\mathcal{C}_{s}$ and
$\mathcal{C}_{t}$. Due to the novel/fine-grained categories only existing in
the target dataset, we have $\mathcal{C}_{s}\neq\mathcal{C}_{t}$. The shared
and target-private new classes are indicated as
$\overline{\mathcal{C}}=\mathcal{C}_{s}\cap\mathcal{C}_{t}$ and
$\hat{\mathcal{C}}=\mathcal{C}_{t}\backslash\overline{\mathcal{C}}$. We call
$\overline{\mathcal{C}}$ existing in both datasets as anchor classes since
they can be used as anchors to facilitate knowledge transfer. The goal is to
train a model jointly on
$\mathcal{D}_{s},\mathcal{D}_{t}^{u},\mathcal{D}_{t}^{l}$ which performs
inferences on data from $\mathcal{P}_{t}$ with labels in $\mathcal{C}_{t}$.

Figure 2: Overview of our proposed method.  We devise the cross-domain and
cross-class distillation module as well as the historical self-distillation
module to transfer the model optimization knowledge from reliable sources by
imposing constraints on the gradient descent steps _w.r.t._ the insufficiently
supervised target domain and new classes.

###  3.2 Characterization of Optimization Trajectory

The technical challenge for the identified problem lies in overcoming both
domain shifts and overfitting incurred by scarce training samples. As
discussed in Section 1, those issues could lead to unreliable navigation
during neural network optimization [20, 10]. Therefore, we propose to distill
the optimization trajectory knowledge from well-annotated sources to
regularize the training dynamics of target domain and new classes.

Recently, NTK theory [29] has been regarded as a tool to dictate the evolution
of neural networks. For a network $f$ parameterized by $\theta$, its training
dynamics can be described by a kernel $K$. Given a training set with $N$
samples, the kernel can be defined as an $N\times N$ matrix with
$K(i,j;\theta)=\nabla_{\theta}f(x^{i};\theta)^{\top}\cdot\nabla_{\theta}f(x^{j};\theta)$,
where $x^{i}$ and $x^{j}$ are two input samples. It indicates the strong
correlations between the network’s optimization trajectory and the gradients
_w.r.t._ its parameters. We thus propose to collect a set of intermediate
gradients during training and capture their statistics to characterize the
optimization trajectory. Specifically, the second-order Taylor expansion shows
that:

$\begin{split}f(x;\theta)=&f(x;\theta^{*})+(\theta-\theta^{*})^{\top}\cdot\nabla_{\theta}f(x;\theta^{*})\cr+&\frac{1}{2}(\theta-\theta^{*})^{\top}\cdot\nabla_{\theta}^{2}f(x;\theta^{*})\cdot(\theta-\theta^{*})\cr+&\mathcal{O}(\left\|\theta-\theta^{*}\right\|^{2}),\end{split}$
(1)

which motivates us to model the first- and second-order derivatives (Hessian)
of $f$ to represent its learning dynamics. We use the mean of gradients to
indicate the first-order derivative and follow [30] to approximate the second-
order derivative with gradient variances.

###  3.3 Cross-Domain and Cross-Class Distillation

Gradient-based optimization algorithms depend on a large number of update
steps supervised by sufficient labeled data [52]. In domain-shifted and small-
data regimes, those algorithms suffer from noisy update signals and
overfitting, without guarantee to converge towards generalizable minima [24].
To this end, we propose to distill the training dynamics from the label-rich
source domain and anchor classes to the target domain and new classes for
external regularization by leveraging their intermediate gradients. We firstly
compute the gradients $\bm{g}$ for each domain and class by backpropagating
the obtained losses. With a general backbone $\mathbf{F}$ for feature
extraction and a task-specific predictor $\mathbf{C}$ for sample-/pixel-wise
classification, the individual gradients can be expressed as:

$\bm{g}:=[\frac{\partial\mathcal{L}(\mathbf{C}(\mathbf{F}(x)),y)}{\partial\theta_{\mathbf{C}_{1}}},\cdots,\frac{\partial\mathcal{L}(\mathbf{C}(\mathbf{F}(x)),y)}{\partial\theta_{\mathbf{C}_{w}}}],$
(2)

where $\mathcal{L}$ is the loss function, $w$ indicates the total number of
parameters in $\mathbf{C}$, $x$ and $y$ are training data and the
corresponding label. For unlabeled samples $\mathbf{x}_{t}^{u}$ in the target
dataset, online pseudo-labels $\tilde{\mathbf{y}}_{t}^{u}$ are used for loss
calculation. Then, as shown in Fig. 2, we aggregate the individual-level
gradients in a memory buffer and retrieve the domain- and class-level gradient
statistics to model the global first- and second-order derivatives. According
to Section 3.2, for gradients _w.r.t._ domain/class $\pi$ collected from a
training set with $n_{\pi}$ samples, we derive their mean and variance as:

$\bm{g}_{\pi}^{m}=\frac{1}{n_{\pi}}\sum\nolimits_{i=1}^{n_{\pi}}\bm{g}_{\pi}^{i},$
(3)
$\bm{g}_{\pi}^{v}=\frac{1}{n_{\pi}-1}\sum\nolimits_{i=1}^{n_{\pi}}(\bm{g}_{\pi}^{i}-\bm{g}_{\pi}^{m})^{2}.$
(4)

Those metrics are measured for each domain and class separately.

However, the gradients obtained from insufficiently annotated domain and
classes are dubious and error-prone [63]. To alleviate the issue, we take
inspiration from recent learning theory findings that stochastic gradients
along the optimization steps are generally low-rank [39], which indicates that
the optimization process is governed by the top gradient eigenspace.
Specifically, we devise a gradient projection approach to suppress the noisy
model update signals in particular to the target domain and new classes by
projecting their gradients on the reliable subspaces identified by the source
domain and anchor classes with sufficient supervision. Our approach involves
three iterative steps, _i.e._ , subspace construction, gradient projection,
and gradient statistics matching. Firstly, to identify the principal
eigenspace, we perform Singular Value Decomposition (SVD) on the aggregated
gradients from the source domain and anchor classes. We denote gradients
collected from source domain, target domain, anchor class, and new class as
$\bm{g}_{s}$, $\bm{g}_{t}$, $\bm{g}_{A}$, and $\bm{g}_{N}$. Given the sets of
gradients $\bm{G}_{s}=[\bm{g}_{s}^{1},\bm{g}_{s}^{2},\cdots,\bm{g}_{s}^{K}]$
and $\bm{G}_{A}=[\bm{g}_{A}^{1},\bm{g}_{A}^{2},\cdots,\bm{g}_{A}^{K}]$ stored
in the memory buffer of volume $K$, we apply SVD:

$\bm{G}_{s}=\bm{U}_{s}\bm{\Sigma}_{s}\bm{V}_{s}^{\top},\qquad\bm{G}_{A}=\bm{U}_{A}\bm{\Sigma}_{A}\bm{V}_{A}^{\top},$
(5)

where $\bm{U}$, $\bm{V}$, and $\bm{\Sigma}$ contain left singular vectors
$\bm{u}^{i}$, right singular vector $\bm{v}^{i}$, and non-negative singular
values $\bm{\sigma}^{i}$, respectively. To capture the overall
characteristics, we use the gradients averaged along all anchor classes to
represent $\bm{G}_{A}$. Next, we adopt the low-rank matrix approximation to
select the top-$r$ significant left singular vectors in $\bm{U}_{s}$ and
$\bm{U}_{A}$ based on the following criteria:

$\left\|(\bm{G}_{s})^{r_{s}}\right\|_{F}^{2}\geq\tau\left\|\bm{G}_{s}\right\|_{F}^{2},\quad\left\|(\bm{G}_{A})^{r_{A}}\right\|_{F}^{2}\geq\tau\left\|\bm{G}_{A}\right\|_{F}^{2},$
(6)

where
$(\bm{G})^{r}=\sum\nolimits_{i=1}^{r}\bm{\sigma}^{i}\bm{u}^{i}\bm{v}^{i\top}$,
$\left\|\cdot\right\|_{F}$ denotes the Frobenius norm, $\tau$ is a threshold
to ensure most information is preserved and is set to 0.98. The principal
subspace and the corresponding projection matrix are thereby constructed as
$\bm{M}_{s}=[\bm{u}_{s}^{1},\bm{u}_{s}^{2},\cdots,\bm{u}_{s}^{r_{s}}]$ and
$\bm{M}_{A}=[\bm{u}_{A}^{1},\bm{u}_{A}^{2},\cdots,\bm{u}_{A}^{r_{A}}]$.
Afterward, all gradients are projected on the identified subspace as shown in
Fig. 2:

$\begin{split}\bm{g}_{s}^{*}=\bm{M}_{s}\bm{M}_{s}^{\top}\bm{g}_{s},&\qquad\bm{g}_{t}^{*}=\bm{M}_{s}\bm{M}_{s}^{\top}\bm{g}_{t},\cr\bm{g}_{A}^{*}=\bm{M}_{A}\bm{M}_{A}^{\top}\bm{g}_{A},&\qquad\bm{g}_{N}^{*}=\bm{M}_{A}\bm{M}_{A}^{\top}\bm{g}_{N}.\end{split}$
(7)

Then following Eq.(3)(4), we minimize the discrepancy between the statistics
(_i.e._ , mean and variance) of projected gradients to distill the learning
dynamics from the source domain and anchor classes to the target domain and
new classes. The overall training objective can be formulated as:

$\begin{split}&\mathop{\mathrm{min}}\limits_{\theta}\ \
\mathcal{L}_{ERM}+\lambda\Big{\\{}\left\|(\bm{g}_{s}^{m*}-\bm{g}_{t}^{m*})\right\|_{2}^{2}+\left\|(\bm{g}_{s}^{v*}-\bm{g}_{t}^{v*})\right\|_{2}^{2}\cr&+\frac{1}{n_{new}}\sum\nolimits_{i=1}^{n_{new}}\big{[}\left\|(\bm{g}_{A}^{m*}-\bm{g}_{Ni}^{m*})\right\|_{2}^{2}+\left\|(\bm{g}_{A}^{v*}-\bm{g}_{Ni}^{v*})\right\|_{2}^{2}\big{]}\Big{\\}}.\end{split}$
(8)

Here $\mathcal{L}_{ERM}$ denotes the empirical risk loss, which is implemented
with cross-entropy loss for classification and Dice loss [48] for
segmentation. $\lambda$ is a balancing coefficient, $n_{new}$ is the number of
new classes in the target domain. It is noted that there exists discrepancy
between the anchor and new classes in semantic concepts, which makes feature
alignment across classes harmful due to negative transfer [64]. However,
aligning class-wise learning dynamics by enforcing the similarity between
their gradients could contribute to lower empirical error on new classes. We
prove this property theoretically in the supplementary material.

###  3.4 Historical Self-distillation

Inferior generalization behaviour is an outstanding challenge for taxonomy
adaptive cross-domain adaptation [21]. When domain shifts and unseen classes
exist, a trained model could only maintain its performance on test samples
sharing multiple similar attributes with the training data but cannot
generalize well to the disparate ones [72]. Motivated by the connection
between flat minima and generalizability [34], we devise the historical self-
distillation module to smooth the optimization path towards a well-
generalizable solution by calibrating gradient descent steps.

To reach flat loss landscapes [6], assuming a local loss function $\psi$ and a
network $f$ parameterized by $\theta$, instead of minimizing the original loss
$\psi(f(x;\theta))$ only, the optimization objective should be:

$\mathop{\mathrm{min}}\limits_{\theta}\
-\log\int_{\left\|\theta-\theta^{*}\right\|\leq\Gamma}\exp(-\psi(f(x;\theta^{*}))-\gamma\left\|\theta-\theta^{*}\right\|)\
d\theta^{*},$ (9)

where $x$ indicates random input samples, $\Gamma$ is defined as the width of
a local parameter valley, $\gamma$ is a balancing weight. In other words, when
$\theta$ and $\theta^{*}$ are neighbouring, the following criteria is required
to be satisfied:

$\left\|\nabla_{\theta}\psi(f(x;\theta))-\nabla_{\theta}\psi(f(x;\theta^{*}))\right\|\leq\beta\left\|\theta-\theta^{*}\right\|,$
(10)

where $\beta$ is a smoothness factor. To this end, we propose to impose
regulations on the optimizer steps by enforcing minima with uniformly-
distributed gradients. Given that the gradients from most iterations are
generally low-rank [39, 2], we exploit past gradients to identify the low-
dimensional subspace which indicates the principal gradient distribution of
local minima and then project current gradients on the constructed subspace to
exclude sharp and noisy signals. Specifically, we collect and store individual
gradients for each iteration along the training procedure, denoted as
$\bm{G}_{It}=[\bm{g}_{It}^{t-T},\bm{g}_{It}^{t-T+1},\cdots,\bm{g}_{It}^{t-2},\bm{g}_{It}^{t-1}]$,
where $t$ is the index of current iteration, $T$ is the size of memory buffer.
Thereafter we perform SVD on the set of gradients $\bm{G}_{It}$ to construct
the historical subspace. Similar to Eq.(5)(6)(7), the top-$r$ left singular
vectors of the decomposed gradient set are concatenated to formulate the
projection matrix $\bm{M}_{It}$. Then for upcoming gradients obtained from
backpropagation in each iteration, we project the gradients on the identified
low-rank subspace and consider them as residuals. The subsequent optimizer
step is conducted with the sum of the original and rectified gradients, as
shown in Fig. 2. Take vanilla stochastic gradient descent as an example, with
mini-batch gradients $\tilde{\bm{g}}$ in each iteration, the parameters
$\theta$ of the optimized network are updated by:

$\theta:=\theta-\frac{1}{\kappa}\cdot\eta\cdot(\bm{M}_{It}\,\bm{M}_{It}^{\top}+\kappa)\cdot\tilde{\bm{g}},$
(11)

where $\eta$ is the learning rate, $\kappa$ is a trade-off parameter. As
training proceeds, the principal subspace and the corresponding projection
matrix are periodically renewed by applying SVD on the recently collected
gradients. The overall training procedure is summarized in the supplementary
material.

Table 1: Comparison results of our proposed method against other state-of-the-art methods for nuclei segmentation and recognition. The metrics with asterisk (*) are calculated on target-private new classes only. The subscripts denote the standard deviations. The best and second-best results are highlighted in bold and brown, respectively. Methods | 5-shot | 10-shot
---|---|---
mF1 | mF1* | mPQ | mPQ* | mF1 | mF1* | mPQ | mPQ*
Sup-only |  28.080.89 |  21.460.36 |  12.350.53 |  12.230.45 |  30.900.66 |  25.230.83 |  14.180.24 |  14.940.32
Multi-task [54] |  33.851.26 |  18.151.08 |  18.580.42 |  10.770.35 |  35.150.94 |  21.290.62 |  19.140.27 |  12.890.34
DANN [18] |  32.761.41 |  16.521.10 |  18.310.66 |  10.400.59 |  35.761.35 |  23.120.88 |  19.700.60 |  14.140.42
CGDM [16] |  35.031.28 |  21.040.99 |  19.590.50 |  13.220.32 |  37.131.57 |  24.221.15 |  19.780.71 |  13.900.40
LETR [45] |  27.781.44 |  15.361.02 |  16.680.54 |  10.030.51 |  34.851.54 |  20.030.73 |  18.180.59 |  11.790.27
FT-CIDA [35] |  30.510.92 |  16.550.80 |  15.970.39 |  9.420.40 |  32.631.21 |  21.080.95 |  17.090.47 |  11.960.49
STARTUP [50] |  37.030.58 |  22.440.76 |  20.400.33 |  13.820.30 | 40.850.50 |  25.920.91 | 23.370.19 |  16.640.24
DDN [27] | 37.920.86 | 23.711.05 | 20.620.57 | 13.870.43 |  40.160.72 | 26.941.08 |  22.790.32 | 16.820.50
TSA [38] |  34.031.17 |  19.551.11 |  18.650.43 |  12.060.37 |  34.271.05 |  22.581.20 |  19.780.61 |  14.240.73
TACS [21] |  36.731.65 |  22.261.29 |  18.130.80 |  12.570.62 |  39.291.48 |  25.841.13 |  22.270.68 |  15.980.56
Ours | 40.260.90 | 27.140.97 | 21.780.49 | 14.960.22 | 43.880.52 | 31.430.78 | 24.810.26 | 19.350.33

###  3.5 Theoretical Analysis

In this section, we motivate our proposed method theoretically and provide
mathematical insights on its merits.

Joint Characterization of Feature and Output Space While our optimization
trajectory distillation approach proposes a novel perspective that differs
from existing works imposing regularization terms in feature and model-output
space [35, 21], those information is still captured and implicitly modeled in
the gradient-based framework. We prove this property for the classification
and segmentation tasks under two commonly adopted loss functions. The detailed
proof is presented in the supplemental material. It illustrates the
superiority of our method as a unified framework that jointly characterizes
the feature and output space as well as the learning dynamics.

Impacts on Generalization Error In our method, we propose to minimize the
discrepancy between the gradient statistics from different domains and classes
in the identified principal subspace. We hereby prove its effectiveness
towards a tighter generalization error bound on the target domain and new
classes [3]. Details can be found in the supplemental material.

##  4 Experiments

###  4.1 Datasets and Experiment Settings

To evaluate the effectiveness of our method and its potential clinical
implications, we conduct experiments on five medical image datasets in regard
to three important down-stream tasks from different clinical scenarios.
Extended experiments on more diverse tasks, including radiology and fundus
analysis, as well as general visual task, are presented in the supplemental
material to substantiate the broader applicability of our method.

Nuclei Segmentation and Recognition Technically, this task can be formalized
as a simultaneous instance segmentation and classification problem [23]. We
evaluate our method by considering PanNuke [17] and Lizard [22] as the source
and target datasets. They contain 481 and 291 visual fields cropped from
whole-slide images with 189,744 and 495,179 annotated nuclei, respectively.
There are six types of nuclei in Lizard and only two of them overlap with the
five classes in PanNuke, which suggests four new classes in the target domain.
We employ the widely used Hover-Net [23] architecture with a standard
ResNet-50 backbone as the base model. For quantitative evaluation, we follow
[22] and report the average F1 and PQ (panoptic quality) scores for all
classes (denoted as mF1 and mPQ) to provide a comprehensive assessment
measuring the accuracy of nuclei detection, segmentation, and classification.
Additionally, we also report the mF1* and mPQ* scores which are solely
computed on new classes in the target domain.

Cancer Tissue Phenotyping In this setting, we perform adaptation from a two-
class discrimination task to an eight-class one on CRC-TP [31] and Kather [33]
datasets. Kather consists of 5000 $150\times 150$ pathology image patches and
includes six novel classes other than the two common classes also existing in
the source domain. The experiments are conducted with ResNet-101 as backbone.
For evaluation, we exploit the accuracy and F1 scores as metrics to indicate
the classification performance. The average scores for all classes (mAcc and
mF1) and particularly the ones on new classes only (mAcc* and mF1*) are
reported.

Skin Lesion Diagnosis We construct a fine-grained taxonomy adaptation setting
from two coarse classes to seven subclasses with HAM10000 [59], which contains
10015 dermatoscopic images sampled from different body structures. The adopted
experiment settings and evaluation metrics are the same as the cancer tissue
phenotyping task. Since the label space of the source and target datasets do
not overlap, we only report the mAcc* and mF1* scores which are computed on
target classes.

Following previous works in UDA [7], for all experiments, each dataset is
randomly split into $80\%/20\%$ for training/testing. To explicitly recognize
the target-private new classes, we sample few (5/10) samples with
corresponding labels to formulate the support set. The remaining target data
is left unlabeled. More details can be found in the supplemental material.

Table 2: Comparison results of our proposed method against other state-of-the-art methods for cancer tissue phenotyping.  Methods | 5-shot | 10-shot
---|---|---
mAcc | mAcc* | mF1 | mF1* | mAcc | mAcc* | mF1 | mF1*
Sup-only |  46.821.30 |  42.261.26 |  44.641.49 |  42.831.08 |  48.521.02 |  45.931.67 |  45.341.45 |  46.151.83
Multi-task [54] |  48.350.78 |  42.021.34 |  45.551.12 |  41.681.63 |  48.560.69 |  45.001.72 |  46.030.64 |  44.461.21
DANN [18] |  47.460.67 |  42.531.22 |  43.081.14 |  43.131.30 |  48.420.48 |  45.311.27 |  46.500.81 |  44.020.64
CGDM [16] |  49.020.82 | 43.531.56 | 46.290.87 | 43.580.69 | 50.540.84 |  44.571.35 | 48.170.66 |  45.230.40
LETR [45] |  45.471.95 |  41.331.47 |  43.171.88 |  41.202.02 |  45.940.56 |  40.131.30 |  41.161.13 |  39.610.90
FT-CIDA [35] |  43.720.88 |  39.060.79 |  40.911.02 |  38.730.84 |  44.650.53 |  38.800.94 |  40.940.96 |  38.961.07
STARTUP [50] | 49.520.63 |  42.931.83 |  43.591.37 |  39.970.59 |  49.400.51 | 46.911.87 |  46.380.70 | 47.851.36
DDN [27] |  47.141.75 |  43.250.99 |  42.922.29 |  40.760.50 |  48.270.67 |  44.081.22 |  46.031.14 |  45.441.18
TSA [38] |  48.221.49 |  42.130.78 |  44.391.84 |  41.921.13 |  47.181.02 |  41.261.10 |  45.341.72 |  44.691.89
TACS [21] |  45.301.07 |  40.281.11 |  44.741.26 |  42.040.95 |  46.951.16 |  43.691.74 |  47.461.51 |  47.511.25
Ours | 50.140.76 | 47.661.54 | 47.411.10 | 46.451.15 | 55.860.98 | 51.271.02 | 52.980.49 | 52.120.94
Table 3: Comparison results of our proposed method against other state-of-the-art methods for skin lesion diagnosis.  Methods | 5-shot | 10-shot
---|---|---
mAcc* | mF1* | mAcc* | mF1*
Sup-only |  35.710.79 |  19.890.22 |  38.500.38 |  22.360.67
Multi-task [54] |  33.801.20 |  19.060.31 |  37.850.40 |  23.320.32
DANN [18] |  37.220.57 |  23.990.92 |  40.060.94 |  24.320.72
CGDM [16] |  41.501.03 |  24.260.80 | 47.220.70 |  24.000.23
LETR [45] |  36.040.42 |  21.390.66 |  40.871.22 |  23.500.49
FT-CIDA [35] |  31.950.77 |  18.130.48 |  34.600.90 |  20.960.88
STARTUP [50] |  43.211.15 | 27.170.92 |  43.881.35 |  25.590.60
DDN [27] |  36.191.05 |  21.500.34 |  42.461.29 |  23.920.28
TSA [38] |  40.411.36 |  24.051.01 |  42.071.51 |  25.441.06
TACS [21] | 45.050.46 |  25.600.65 |  46.730.64 | 25.960.79
Ours | 50.790.69 | 30.050.81 | 52.140.98 | 30.520.83

###  4.2 Results

We compare our method with state-of-the-art approaches for cross-domain
adaptation, including UDA methods DANN [18] and CGDM [16] which do not
consider the taxonomic inconsistency, as well as LETR [45], FT-CIDA[35],
STARTUP [50], DDN [27], TSA [38], and TACS [21] which are designed for both
domain shifts and new class learning. For UDA methods, we jointly perform
domain alignment and train the model with the labeled target data. In
addition, we conduct comparisons with multi-task learning approach [54], which
trains the model on all domains and label sets simultaneously. “Sup-only”
indicates the baseline where only the supervised loss on the labeled target
dataset is used for training.

Table 1 shows the comparison results on the cross-domain nuclei recognition
and segmentation task. All methods adopt the same ResNet-50-HoverNet as base
model following [23]. It can be observed that our proposed method outperforms
the state-of-the-art cross-domain adaptation approaches by a large margin. We
also notice an important point that compared with the “Sup-only” baseline,
most competing methods fail to attain better performance on the metrics
specific to new classes (_i.e.,_ mF1* and mPQ*). For example, under the 5-shot
setting, despite a 4.68$\%$ improvement is achieved by DANN on mF1, it
achieves inferior performance on mF1* which is 4.94$\%$ worse than “Sup-only”.
This phenomenon is incurred by the failure of cross-class generalization. In
contrast, our method yields substantial improvements on those new class-
specific metrics, which indicates the superior generalizability of our method.
The observation is further verified through visual comparisons in the
supplementary material.

The overall quantitative results on the cancer tissue phenotyping and skin
lesion diagnosis tasks are presented in Table 2 and 3, respectively. All
methods are implemented based on the ResNet-101 backbone. Under each few-shot
setting, our method shows higher classification scores compared with previous
approaches. In particular, we achieve 0.62$\%$$\sim$5.32$\%$ and
1.12$\%$$\sim$4.81$\%$ improvements against the second-best results in terms
of mAcc and mF1 for cancer tissue phenotyping, as well as
4.92$\%$$\sim$7.58$\%$ and 2.88$\%$$\sim$4.93$\%$ improvements in terms of
mAcc* and mF1* for skin lesion diagnosis. The higher accuracy consistently
attained by our method under different cross-domain adaptation settings and
clinical tasks is attributed to the strong generalizability brought by the
proposed optimization trajectory distillation approach.

###  4.3 Analysis and Discussion

Ablation of Key Components Table 4 shows the ablation study results to
illustrate the effectiveness of key components. “projection” denotes the
gradient projection approach introduced in Section 3.3. For the “ _w/o_
projection” ablation, we perform cross-domain and cross-class distillation by
directly minimizing the statistics discrepancy between unrectified gradients
$(g_{s},g_{t})$ and $(g_{A},g_{N})$ in Eq. (7). It is observed that each
component is indispensable and could improve the overall performance, which
elaborates the importance of external navigation for learning target domain
and new classes in network training. Interestingly, we notice that the impacts
of cross-domain and cross-class distillation could be disparate across tasks.
For example, cross-class distillation brings higher performance gain compared
with cross-domain distillation on the cancer tissue phenotyping benchmark,
while the situation is reversed for skin lesion diagnosis. It is caused by the
different natures of each task. Class-imbalance is a serious issue for skin
lesion diagnosis in that the class distribution is highly biased [59], which
compromises the effectiveness of the cross-class distillation module.
Differently, for cancer tissue phenotyping, classes are uniformly distributed
[33] and could hence ensure the representativity of anchor classes and their
exemplarity to be distilled to new classes.

Table 4: Key component analysis of our method on three cross-domain adaptation benchmarks under 10-shot scheme. The best results are highlighted in bold.  Methods | Nuclei | Cancer | Skin
---|---|---|---
mF1 | mF1* | mPQ | mPQ* | mAcc | mAcc* | mF1 | mF1* | mAcc* | mF1*
Full Framework | 43.88 | 31.43 | 24.81 | 19.35 | 55.86 | 51.27 | 52.98 | 52.12 | 52.14 | 30.52
_w/o_ cross-domain | 39.96 | 30.08 | 22.29 | 18.10 | 52.81 | 49.93 | 51.62 | 50.64 | 44.34 | 25.28
_w/o_ cross-class | 42.30 | 27.82 | 23.76 | 17.05 | 48.90 | 47.22 | 49.85 | 44.76 | 49.06 | 28.24
_w/o_ historical | 41.54 | 30.47 | 23.00 | 17.59 | 50.17 | 48.05 | 48.19 | 45.12 | 45.83 | 25.97
_w/o_ projection | 40.06 | 28.80 | 22.10 | 17.31 | 53.58 | 45.78 | 49.01 | 49.20 | 47.29 | 28.02

Hyperparameter Sensitivity To investigate the effect of hyperparameters
$\lambda$ in Eq. (LABEL:equ:objective) and $\kappa$ in Eq. (11), we present
the performance comparison by varying the choices of their values for cancer
tissue phenotyping, as shown in Fig. 3. We only change one hyperparameter
while keeping the other fixed at a time. The experimental results show that
our framework is quite stable for hyperparameters in a wide interval, yet
setting $\lambda$ and $\kappa$ to a large value is detrimental to the
classification accuracy. Specifically, $\lambda$ is a balancing coefficient
between class discriminativeness and gradient alignment. When it is too large,
the trained model is biased to matching learning dynamics and would lose
essential semantic knowledge. For $\kappa$, it controls the proportions of
unrectified gradients. Its large value indicates that most original gradients
are preserved, which would incur noisy optimization steps and loss of
flatness. Therefore, we decide to set $\lambda=10$ and $\kappa=100$. It
achieves the best performance under most metrics.

Figure 3: Performance comparison between different choices of hyperparameters
$\lambda$ and $\kappa$ on the cancer tissue phenotyping benchmark under
10-shot scheme.

Impacts on Optimization Trajectory and Model Flatness To better understand the
effectiveness of our method, we perform in-depth analysis about its impacts on
model’s training trajectory and flatness of the converged local minima.
Firstly, as shown in Fig. 4(a), we visualize the optimization paths of our
method and the multi-task baseline on the test accuracy surface with a contour
plot. The details of plot procedure can refer to [28]. It can be observed that
the optimization of the multi-task baseline fluctuates sharply and is
restricted in a sub-optimal area with a relatively low test accuracy. This is
induced by the biased navigation in the course of training due to domain
shifts and category gap. With the devised dual-stream optimization trajectory
distillation module, our method could by contrast suppress the noisy update
signals and proceed consistently towards the optimum.

Besides, we perform analysis about the solutions found by our method and the
baseline in terms of flatness [5]. Specifically, the local flatness of a model
parameter $\theta$ is quantified by the expected changes of test accuracy
between $\theta$ and a neighbouring $\theta^{*}$ with perturbation rate
$\varrho$:
$\mathcal{F}_{\varrho}(\theta)=\mathbb{E}_{\left\|\theta^{*}\right\|=(1+\varrho)\left\|\theta\right\|}\big{[}\rm{Acc}(\theta^{*})-\rm{Acc}(\theta)\big{]}$.
We approximate the expected value by Monte-Carlo sampling over 50 samples. As
demonstrated in Fig. 4(b), comparing with the baseline approach, our method
shows stronger robustness against model parameter perturbation. It proves that
our method successfully converges to minima with better local flatness and
thus possesses superior generalizability [34].

Figure 4: Further analysis of our proposed method. (a) We plot the
optimization trajectory of our method and the multi-task baseline on the test
accuracy surface. (b) We visualize the local flatness of our method and the
baseline via test accuracy gap by varying the model parameter perturbation
rate. Each point is computed by Monte-Carlo approximation over 50 random
samples. All the experiments are conducted on the cancer tissue phenotyping
benchmark under 10-shot scheme.

##  5 Conclusion

In this paper, we study the taxonomy adaptive cross-domain adaptation
paradigm, which allows incoherent label sets between source and target
datasets. To jointly address the data distribution bias and category gap, we
propose a unified framework which performs cross-domain and cross-class
optimization trajectory distillation to calibrate the learning dynamics of
insufficiently annotated domain and classes. Historical self-distillation is
further devised to attain a well-generalizable solution via regulating
gradient distributions. Extensive experiments on multiple benchmarks with
different task contexts demonstrate the effectiveness and generalization
capability of our method. In the future, we will extend this work to further
discover the underlying mechanism of optimization trajectory distillation for
model generalization and evaluate its potential in more adaptation scenarios.

Supplementary Material: Taxonomy Adaptive Cross-Domain Adaptation in Medical
Imaging via Optimization Trajectory Distillation

In this supplementary, we provide additional illustration of our proposed
method and proofs to support theoretical analysis, as well as dataset and
implementation details. Extended experiments and analysis are performed to
further verify the effectiveness and robustness of our method.

##  1 Overall Traininig Procedure

Input: Source dataset $\mathcal{D}_{s}$, target dataset $\mathcal{D}_{t}$,
number of iterations $I$

Output: Optimized model parameters $\theta$

1 Init: Gradient memory buffer $\bm{G}_{s},\bm{G}_{A},\bm{G}_{It}$

2 for  _$i\leftarrow 1$ to $I$_ do

3 $\\{(\mathbf{x}_{s},\mathbf{y}_{s})\\}\leftarrow$ Image-label pairs sampled
from$(\mathcal{D}_{s})$

4
$\\{(\mathbf{x}_{t}^{u})\\},\\{(\mathbf{x}_{t}^{l},\mathbf{y}_{t}^{l})\\}\leftarrow$
Sample from$(\mathcal{D}_{t})$

5 Generate online pseudo-label $\tilde{\mathbf{y}}_{t}^{u}$

6 $\\{(\mathbf{x}_{t},\mathbf{y}_{t})\\}\leftarrow$
$\\{(\mathbf{x}_{t}^{u},\tilde{\mathbf{y}}_{t}^{u}),(\mathbf{x}_{t}^{l},\mathbf{y}_{t}^{l})\\}$

7 Backpropagate gradients $\bm{g}_{s},\bm{g}_{t},\bm{g}_{A},\bm{g}_{N}$ by
Eq.(2)

8 Cross-domain/class distillation:

9 Update $\bm{G}_{s}$ and $\bm{G}_{A}$ with $\bm{g}_{s}$ and $\bm{g}_{A}$

10 if  _$\bm{G}_{s}$ $\rm{and}$ $\bm{G}_{A}$ $\rm{are\ full}$_ then

11 Apply SVD to identify the principal subspace and form the corresponding
projection matrix $\bm{M}_{s},\bm{M}_{A}$ by Eq.(5)(6)

12 Clear $\bm{G}_{s}$ and $\bm{G}_{A}$

13 end if

14 Perform gradient projection by Eq.(7)

15 Compute the overall training objective $\mathcal{L}$ by Eq.(8)

16 Temporal self-distillation:

17 Update $\bm{G}_{It}$ with $\bm{g}_{It}$

18 if  _$\bm{G}_{It}$ $\rm{is\ full}$_ then

19 Form the projection matrix $\bm{M}_{It}$ by Eq.(5)(6)

20 Clear $\bm{G}_{It}$

21 end if

22 Compute the mini-batch gradients
$\tilde{\bm{g}}\leftarrow\nabla_{\theta}\mathcal{L}(\theta)$

23 Update model parameters $\theta$ by Eq.(11)

24 end for

return $\theta$

Algorithm 1 Training Procedure for Optimization Trajectory Distillation

We summarize the workflow of our proposed method in Algorithm 1. The referred
equation can be found in the main text.

##  2 Details for Theoretical Analysis

###  2.1 Joint Characterization of Feature and Output Space

Here, we prove that in our gradient-based method, information in feature space
and model-output space can be jointly modeled implicitly, which demonstrates
the superiority of our method as a unified framework that jointly
characterizes the feature and output space as well as the learning dynamics.
We illustrate this property under cross-entropy loss and Dice loss.

Specifically, for the task-specific prediction layer, which is implemented as
a convolutional layer with $1\times 1$ kernel, its function can be
mathematically expressed as:

$\bm{u}=\bm{W}^{\top}\bm{z}+\bm{b},$ (12)

where $\bm{z}\in\mathbb{R}^{B\times m\times h\times w}$ denotes the input
feature maps with batch size $B$, channel number $m$, and spatial size
$h\times w$. $\bm{W}\in\mathbb{R}^{m\times c}$ is the convolutional kernel
matrix with input channel $m$ and output channel $c$.
$\bm{b}\in\mathbb{R}^{c}$ indicates the bias tensor.
$\bm{u}\in\mathbb{R}^{B\times c\times h\times w}$ is the logit predictions.

We first consider the gradients for network parameter $\theta$ by
backpropagating the cross entropy (CE) loss $\mathcal{L}_{CE}$:

$\min_{\theta\sim[\bm{W},\,\bm{b}]}\,-\sum_{n=1}^{p}\sum_{i=1}^{c}\bm{y}_{i}^{n}\log\frac{e^{\bm{u}_{i}^{n}}}{\sum_{j=1}^{c}e^{\bm{u}_{j}^{n}}},$
(13)

where $p=B\times h\times w$ is the total number of pixels,
$\bm{y}\in\mathbb{R}^{B\times c\times h\times w}$ is the one-hot pixel-wise
class label. Since our focus is on feature space $\bm{z}$ and output space
$\bm{u}$, without loss of generality, we set $\bm{y}$ to follow the uniform
class distribution that $\bm{y}=[1/c,\,1/c,...,1/c]$. Then, for each pixel,
the derivative of the CE loss can be formulated as:

$\begin{split}&\frac{\partial\mathcal{L}_{CE}}{\partial\theta}\bigg{|}_{\theta=[\theta_{1},\,\theta_{2},\,...,\,\theta_{c}]}=\frac{\partial\mathcal{L}_{CE}}{\partial\bm{u}}\bigg{|}_{\bm{u}=[\bm{u}_{1},\,\bm{u}_{2},\,...,\,\bm{u}_{c}]}\cdot\frac{\partial\bm{u}}{\partial\theta}\\\
&=-\frac{1}{c}\cdot\frac{\partial\,(\sum_{i=1}^{c}\log\frac{e^{\bm{u}_{i}}}{\sum_{j=1}^{c}e^{\bm{u}_{j}}})}{\partial\bm{u}}\cdot\frac{\partial\bm{u}}{\partial\theta}\\\
&=-\frac{1}{c}\cdot\frac{\partial\,(\sum_{i=1}^{c}\bm{u}_{i}-c\cdot\log{\sum_{j=1}^{c}e^{\bm{u}_{j}}})}{\partial\bm{u}}\cdot\frac{\partial\bm{u}}{\partial\theta}\\\
&=-\frac{1}{c}\cdot(1-c\cdot\frac{[e^{\bm{u}_{1}},\,e^{\bm{u}_{2}},\,...,\,e^{\bm{u}_{c}}]}{\sum_{j=1}^{c}e^{\bm{u}_{j}}})\cdot\bm{z}.\end{split}$
(14)

Then we summarize the gradient magnitudes over all pixels and channels:

$\begin{split}&\sum_{i=1}^{c}\lVert\frac{\partial\mathcal{L}_{CE}}{\partial\theta_{i}}\rVert=\sum_{n=1}^{p}\sum_{i=1}^{c}\lVert\frac{\partial\mathcal{L}_{CE}}{\partial\bm{u}_{i}^{n}}\cdot\frac{\partial\bm{u}_{i}^{n}}{\partial\theta_{i}}\rVert\\\
&=-\frac{1}{c}\cdot(\underbrace{\sum_{n=1}^{p}\sum_{i=1}^{c}\lVert
1-c\cdot\frac{e^{\bm{u}_{i}^{n}}}{\sum_{j=1}^{c}e^{\bm{u}_{j}^{n}}}}_{\mathrm{output\,space}}\rVert)\cdot\underbrace{(\sum_{n=1}^{p}\lVert\bm{z}^{n}\rVert)}_{\mathrm{feature\,space}}.\end{split}$
(15)

The result indicates that the gradients of CE loss characterize the
information in both feature space and output space.

Similarly, for Dice loss $\mathcal{L}_{Dice}$:

$\min_{\theta\sim[\bm{W},\,\bm{b}]}\,\sum_{n=1}^{p}\sum_{i=1}^{c}(1-\frac{2\bm{y}_{i}^{n}\mathtt{act}(\bm{u}_{i}^{n})}{\bm{y}_{i}^{n}+\mathtt{act}(\bm{u}_{i}^{n})}).$
(16)

Here we omit the details of softmax layer and use $\mathtt{act}$ to denote the
activation function for simplicity. When $c$ is large, the pixel-wise
derivative can be approximated by:

$\begin{split}&\frac{\partial\mathcal{L}_{Dice}}{\partial\theta}\bigg{|}_{\theta=[\theta_{1},\,\theta_{2},\,...,\,\theta_{c}]}\\\
&=\frac{\partial\mathcal{L}_{Dice}}{\partial\mathtt{act}(\bm{u})}\cdot\frac{\partial\mathtt{act}(\bm{u})}{\partial\bm{u}}\cdot\frac{\partial\bm{u}}{\partial\theta}\\\
&\approx\bm{\xi}\cdot\frac{[e^{\bm{u}_{1}},\,e^{\bm{u}_{2}},\,...,\,e^{\bm{u}_{c}}]}{[\mathtt{act}(\bm{u}_{1})^{2},\,\mathtt{act}(\bm{u}_{2})^{2},\,...,\,\mathtt{act}(\bm{u}_{c})^{2}]}\cdot\bm{z},\end{split}$
(17)

where $\bm{\xi}$ is a constant term. Its gradient magnitudes can be written
as:

$\begin{split}\sum_{i=1}^{c}\lVert\frac{\partial\mathcal{L}_{Dice}}{\partial\theta_{i}}\rVert&=\sum_{n=1}^{p}\sum_{i=1}^{c}\lVert\frac{\partial\mathcal{L}_{Dice}}{\partial\mathtt{act}(\bm{u}_{i}^{n})}\cdot\frac{\partial\mathtt{act}(\bm{u}_{i}^{n})}{\partial\bm{u}_{i}^{n}}\cdot\frac{\partial\bm{u}_{i}^{n}}{\partial\theta_{i}}\rVert\\\
&\approx\bm{\xi}\cdot(\underbrace{\sum_{n=1}^{p}\sum_{i=1}^{c}\lVert\frac{e^{\bm{u}_{i}^{n}}}{\mathtt{act}(\bm{u}_{i}^{n})^{2}}}_{\mathrm{output\,space}}\rVert)\cdot\underbrace{(\sum_{n=1}^{p}\lVert\bm{z}^{n}\rVert)}_{\mathrm{feature\,space}},\end{split}$
(18)

which proves that the proposition also holds true for Dice loss.

###  2.2 Impacts on Generalization Error

In this section, we prove the effectiveness of our method towards a tighter
generalization error bound on the target domain and novel classes.

Firstly, we analyze the underlying mechanism for the cross-domain distillation
module. Let $\mathcal{H}$ be a hypothesis space of VC-dimension $d$, for
$h\in\mathcal{H}$, the correlations between the error on the target domain and
the distance in gradient space across domains are established as [3, 19]:

$\begin{split}\epsilon&{}_{T}(h)\leq\hat{\epsilon}_{S}(h)+\frac{4}{n_{l}}\sqrt{(d\log\frac{2en_{l}}{d}+\log\frac{4}{\delta})}+\Lambda\\\
&+\mathbf{Div}_{\nabla}\boldsymbol{(\tilde{\mathcal{U}}_{S},\tilde{\mathcal{U}}_{T})}+4\sqrt{\frac{4\log(2n_{u})+\log(\frac{4}{\delta})}{n_{u}}},\end{split}$
(19)

with probability at least $1-\delta$. Here $\epsilon_{T}$ and
$\hat{\epsilon}_{S}$ represent the true and empirically estimated error of the
target and source domain, respectively.
$Div_{\nabla}(\tilde{\mathcal{U}}_{S},\tilde{\mathcal{U}}_{T})$ is the
distance between data distributions $\tilde{\mathcal{U}}_{S}$ and
$\tilde{\mathcal{U}}_{T}$ in gradient space. $n_{l}$ and $n_{u}$ denote the
number of labeled and unlabeled samples. $\Lambda$, $\delta$, and $e$ are
constants. It implies that constraining the gradient descent trajectory of the
target domain to approximate the source domain’s, which reduces
$Div_{\nabla}(\tilde{\mathcal{U}}_{S},\tilde{\mathcal{U}}_{T})$, could lead to
lower cross-domain generalization error.

Furthermore, we demonstrate that the cross-class distillation module
contributes to lower empirical error on novel classes from the multi-task
learning perspective. Suppose that $\mathcal{L}$ is the empirical training
loss and $\nabla_{\theta}\mathcal{L}_{q}(\theta)$ denotes its derivative
_w.r.t._ class $q$. Given a set of anchor classes $\\{a_{i}\\}_{i=1}^{A}$ and
a novel class $q$, with the first-order Taylor expansion, we have:

$\mathcal{L}_{q}(\theta-\mu\cdot\Delta\theta^{*})=\mathcal{L}_{q}(\theta)-\mu\cdot\nabla_{\theta}\mathcal{L}_{q}(\theta)\cdot\Delta\theta^{*}+\mathcal{O}(\mu),$
(20)

where the optimization step $\Delta\theta^{*}$ is characterized by
$\sum_{i=1}^{A}\nabla_{\theta}\mathcal{L}_{a_{i}}(\theta)+\nabla_{\theta}\mathcal{L}_{q}(\theta)$,
$\mu$ is a small value. Then:

$\begin{split}\mathcal{L}_{q}(\theta-\mu\cdot&\Delta\theta^{*})-\mathcal{L}_{q}(\theta)=-\mu\cdot\Big{\\{}\left\|\nabla_{\theta}\mathcal{L}_{q}(\theta)\right\|^{2}\cr+&\sum_{i=1}^{A}\big{[}\boldsymbol{\nabla_{\theta}\mathcal{L}_{q}(\theta)\cdot\nabla_{\theta}\mathcal{L}_{a_{i}}(\theta)}\big{]}\Big{\\}}+\mathcal{O}(\mu).\end{split}$
(21)

It indicates that by enforcing the similarity between the gradients _w.r.t._
novel and anchor classes, we could drive the model to reduce the empirical
loss on novel classes along optimization and thereby attain a well-
generalizable solution.

##  3 Implementation Details

###  3.1 Nuclei Segmentation and Recognition

####  3.1.1 Datasets and Preprocessing

Accurate detection, segmentation, and classification of nuclei serve as
essential prerequisites for various clinical and research studies within the
digital pathology field [23]. Inconsistent taxonomy for nuclei categorization
is common across different institutes, which results in the unmatched label
sets among datasets. In this regard, we use PanNuke [17] and Lizard [22] as
the source and target dataset, respectively. PanNuke contains 481 visual
fields cropped from whole-slide images along with 189,744 annotated nuclei. It
follows a categorization schema where nuclei are divided into five classes,
including neoplastic, non-neoplastic epithelial, inflammatory, connective, and
dead cells. We discard the “dead” class as it does not exist in most image
patches. To ensure the dataset has a uniform data distribution, we use all
images from the breast tissue to formulate the source dataset. Lizard consists
of 291 image regions with an average size of $1016\times 917$ pixels from the
colon tissue and annotates 495,179 nuclei. It adopts a categorization schema
different to PanNuke that there are six classes in total, _i.e._ , neutrophil,
eosinophil, plasma, lymphocyte, epithelial, and connective cells. We use the
Dpath subset as the target dataset. For preprocessing, all the visual fields
with divergent size are randomly cropped into image patches of $128\times 128$
pixels. CutMix [70] is used to augment the target dataset.

####  3.1.2 Network Architectures and Parameter Settings

We employ the widely used Hover-Net [23] architecture with a standard
ResNet-50 backbone as the base model. The optimizer is Adam with a learning
rate of $1e-4$ and $(\beta_{1},\beta_{2})=(0.9,0.999)$, and the batch size is
set as 4. To supervise the classification and segmentation branches, we adopt
a combined loss of $\rm{CrossEntropyLoss+DiceLoss}$. $\lambda$ in Eq.(8) and
$\kappa$ in Eq.(11) are empirically set to 1000 and 10, respectively.

####  3.1.3 Evaluation Metrics

F1 score is a popular metric to evaluate classification performance. It
measures both precision and recall harmonically. We report the class-averaged
score to indicate the overall accuracy. Panoptic quality (PQ) [23] is a
unified metric for the instance segmentation task which models the quality of
both detection and segmentation results concurrently:

$\mathcal{PQ}=\underbrace{\frac{\rm{TP}}{\rm{TP}+\frac{1}{2}\rm{FP}+\frac{1}{2}\rm{FN}}}_{\rm{Detection\
Quality}}\cdot\underbrace{\frac{\sum_{(y,\hat{y})\in\rm{TP}}IoU(y,\hat{y})}{\rm{TP}}}_{\rm{Segmentation\
Quality}}.$ (22)

where $\rm{TP,FP,FN}$ are the true positive, false positive, and false
negative detection predictions, respectively. $(y,\hat{y})$ represents the
pair of ground truth and predicted segmentation mask. $IoU$ is the
intersection over union score.

###  3.2 Cancer Tissue Phenotyping

####  3.2.1 Datasets and Preprocessing

Identifying distinct tissue phenotypes is an essential step towards systematic
profiling of the tumor microenvironment in pathological examination [31].
Previous works are mostly limited to the discrimination of two classes of
tissue: tumor and stroma [41], while recent studies argue that recognizing
more heterogeneous tissues brings clinical value [33]. We therefore propose to
perform adaptation from a dataset with only two categories of tissue to
another dataset with several novel classes. In particular, we select images of
tumor and stroma tissue from the CRC-TP [31] to form the source dataset. CRC-
TP contains 20 H$\rm{\&}$E-stained colorectal cancer (CRC) slides obtained
from 20 different patients. Region-level tissue phenotype annotations are
provided by expert pathologists. To ensure a unique category label can be
assigned to each image, patches are extracted at $20\times$ magnification with
the size of $150\times 150$ pixels. The patch-wise tissue phenotypes are
decided based on the majority of their content. Kather [33] is then regarded
as the target dataset. It consists of 5000 $150\times 150$ pathology image
patches sampled from 10 anonymized CRC tissue slides. Other than tumor and
stroma tissue, Kather includes six novel tissue types, _i.e._ , complex
stroma, immune cells, debris, normal mucosal glands, adipose tissue,
background, and thus poses an 8-class classification problem.

####  3.2.2 Network Architectures and Parameter Settings

For experiments, we employ ResNet-101 as the image encoder and thereupon add
two classification heads on top to perform 2-class and 8-class discrimination,
respectively. During training, cross-entropy loss and Adam optimizer with
learning rate $1e-4$ are used to optimize the model with a batch size of 4.
$\lambda$ and $\kappa$ are set to 10 and 100.

Figure 5: Visual comparisons with other methods on the nuclei segmentation and
recognition benchmark.

###  3.3 Skin Lesion Diagnosis

####  3.3.1 Datasets and Preprocessing

Automatic fine-grained skin lesion recognition remains a global challenge in
dermatology. By taking a step further than the basic benign/malignant
differentiation, identifying the specific subtypes of lesions demonstrates
significant diagnostic value [58]. We hereby assign a benign/malignant
discrimination dataset as the source domain, and a fine-grained multi-disease
dataset as the target domain. HAM10000 is a dermatoscopic image dataset
collected from different populations and modalities [59]. After preprocessing
procedures including histogram correction, sample filtering, and center crop,
10015 dermatoscopic images with lesions of seven diagnostic categories in
total are provided. It contains four subtypes of benign lesions (melanocytic
nevi (NV), benign keratinocytic lesions (BKL), dermatofibromas (DF), and
vascular lesions (VASC)) and three subtypes of malignant ones (melanomas
(MEL), basal cell carcinomas (BCC), and actinic keratoses intraepithelial
carcinomas (AKIEC)). We use the face subset with only coarse two-class
annotations as the source domain and the lower extremity subset with fine-
grained seven-class annotations as the target domain. All images are randomly
cropped to the size of $160\times 160$ pixels before being forwarded to the
network.

###  3.4 Overall Experiment Settings

For all experiments, we implement our method with Pytorch and conduct training
on a NVIDIA GeForce RTX 3090 GPU with 24GB of memory. Gradient backpropagation
is performed for each mini-batch using BackPACK [12]. Following previous works
in UDA [7], each dataset is randomly split into $80\%/20\%$ as the
training/test sets. For novel classes in the target dataset, we sample few
(5/10) samples with corresponding labels to formulate the support set. The
remaining target data is left unlabeled. Data augmentation techniques such as
rotation and horizontal/vertical flip are employed during training. Please
refer to the source code for more details.

It is noted that although from the technical perspective, skin lesion
diagnosis is a multi-class classification problem similar to cancer tissue
phenotyping, they differ largely in the task context. Specifically, skin
lesion diagnosis is more like an object recognition task where its decision is
dominated by the local attributes of lesions, while cancer tissue phenotyping
relies on the global structure of the whole pathology images, instead of
focusing on a salient object.

Figure 6: Qualitative visualizations of our proposed method on the cancer tissue phenotyping and skin lesion diagnosis benchmark with t-SNE plot. Varied colours of points indicate the samples of different classes. Table 5: Comparison results of our proposed method against other state-of-the-art methods for nuclei segmentation and recognition with 30-shot labeled target samples. The best and second-best results are highlighted in bold and brown, respectively. Methods | 30-shot
---|---
mF1 | mF1* | mPQ | mPQ*
Sup-only |  43.780.56 |  36.730.34 |  21.830.54 |  21.270.38
Multi-task [54] |  43.231.02 |  30.391.37 |  23.550.49 |  17.780.72
DANN [18] |  41.701.41 |  28.251.63 |  22.380.57 |  15.990.80
CGDM [16] |  43.820.98 |  34.360.91 |  24.920.34 |  20.480.26
LETR [45] |  40.041.13 |  25.800.82 |  21.960.70 |  14.890.44
FT-CIDA [35] |  40.590.87 |  32.950.54 |  22.370.58 |  19.340.30
STARTUP [50] | 49.000.74 |  36.321.10 | 28.780.46 |  22.570.43
DDN [27] |  48.240.79 |  36.770.85 |  27.900.61 |  22.640.28
TSA [38] |  44.960.60 |  33.130.97 |  25.690.91 |  21.400.63
TACS [21] |  47.550.81 | 37.381.55 |  28.470.83 | 23.181.04
Ours | 51.690.48 | 41.630.65 | 29.420.55 | 25.340.67
Table 6: Comparison results of our proposed method against other state-of-the-art methods on three diverse tasks. Methods | Radiology | Fundus | OfficeHome
---|---|---|---
mF1 | mF1* | mF1 | mF1* | mF1 | mF1*
Baseline | 41.87 | 19.20 | 41.03 | 29.07 | 44.87 | 46.54
CIDA [33] | 42.55 | 23.48 | 39.90 | 27.32 | 43.29 | 42.80
TACS [19] | 46.36 | 22.13 | 44.84 | 32.68 | 42.25 | 47.71
Ours | 49.23 | 26.54 | 46.26 | 37.71 | 50.08 | 54.67
Table 7: Ablation study by evaluating the performance gain of adding each key component. We start from the multi-task baseline [54] and add components cumulatively. Experiments are conducted on the nuclei segmentation and recognition benchmark.  Methods | 5-shot | 10-shot
---|---|---
mF1 | mF1* | mPQ | mPQ* | mF1 | mF1* | mPQ | mPQ*
Baseline | 33.85 | 18.15 | 18.58 | 10.77 | 35.15 | 21.29 | 19.14 | 12.89
+cross-domain | 36.56 | 21.41 | 20.09 | 12.41 | 38.94 | 23.78 | 21.93 | 13.95
+cross-class | 36.40 | 22.84 | 20.15 | 13.30 | 40.52 | 27.26 | 22.78 | 16.13
+historical | 37.38 | 24.90 | 19.71 | 13.17 | 40.06 | 28.80 | 22.10 | 17.31
+projection | 40.26 | 27.14 | 21.78 | 14.96 | 43.88 | 31.43 | 24.81 | 19.35

##  4 Additional Experiment Results

###  4.1 Visualization

We provide additional qualitative results on the three benchmarks. The
comparison results shown in Fig. 5 demonstrate the superiority of our method
to detect each nucleus and delineate its boundary, as well as differentiating
nuclei of various types with their detailed biological features. The t-SNE
visualization in Fig. 6 shows that our method could discover the underlying
embedding structures of various classes even with very limited labeled data.

###  4.2 Results with More Annotations

In this section, we compare our method with previous state-of-the-art
approaches for cross-domain adaptation when more labeled samples are available
in the target domain. The results for nuclei segmentation and recognition with
30 labeled target samples are shown in Table 5. The overall improvements of
our method under this setting are consistent with previous experiment results.
It validates the effectiveness of our method under different levels of
support.

Figure 7: Performance comparison between different choices of hyperparameters $K$, $T$ and $\tau$ on the cancer tissue phenotyping benchmark under 10-shot scheme. Table 8: Comparisons of our proposed method against other state-of-the-art methods for nuclei segmentation and recognition by averaging the results from 10 random selections of support sets in the target domain. The best results are highlighted in bold.  Methods | 5-shot | 10-shot | 30-shot
---|---|---|---
mF1 | mF1* | mPQ | mPQ* | mF1 | mF1* | mPQ | mPQ* | mF1 | mF1* | mPQ | mPQ*
Multi-task [54] | 30.04 | 16.17 | 16.51 | 9.20 | 33.58 | 20.73 | 18.45 | 12.40 | 39.52 | 28.16 | 22.21 | 17.25
CGDM [16] | 33.30 | 19.88 | 19.08 | 12.46 | 37.63 | 26.69 | 20.61 | 15.21 | 42.53 | 35.11 | 23.96 | 20.82
STARTUP [50] | 35.91 | 21.28 | 19.66 | 13.53 | 39.10 | 24.65 | 21.77 | 15.34 | 45.71 | 32.38 | 26.41 | 20.33
Ours | 37.73 | 25.36 | 20.18 | 14.29 | 44.04 | 32.52 | 24.70 | 19.49 | 49.12 | 40.17 | 28.44 | 25.31

###  4.3 Extended Experiments on Diverse Tasks

We further evaluate our method on two medical image tasks beyond pathology
analysis and one general visual recognition task, where the medical image
tasks include pneumonia screening in radiology and diabetic retinopathy
grading in fundus photography. For radiology analysis, we adopt covid-kaggle
[51] and Chest-Montreal [11] for TADA from normal/pneumonia coarse screening
to fine-grained pneumonia diagnosis. For fundus, DDR [37] and APTOS19 [32] are
used to construct a TADA setting with two novel classes (grade level 3, 4). In
these settings, distribution shifts exist across domains due to differences in
image acquisition protocols among multiple cohorts. For general visual task,
we adopt OfficeHome [61] and evaluate on “Artist” and “Real-world” domains.
Experiments are conducted in 10-shot regime and the corresponding results are
presented in Table 6. Through comparison with SOTA methods, the effectiveness
and broader applicability of our method are proved.

###  4.4 Extended Key Component Analysis

In Table 7, we demonstrate the effectiveness of our method’s key components.
It complements the ablation study performed in the main text from a cumulative
perspective. From the results, we observe that all the proposed modules are
beneficial for improving the cross-domain adaptation performance. In
particular, the employment of the cross-class distillation module contributes
to a significant performance gain for target-private classes. For instance, it
attains $3.48\%$ and $2.18\%$ improvements in terms of mF1* and mPQ* under
10-shot scheme. It verifies the effectiveness of our method to perform
optimization trajectory distillation across domains and classes towards strong
model generalization.

###  4.5 Extended Hyperparameter Sensitivity Analysis

We further analyze the choices of hyperparameters and their impacts on model
performance in Fig. 7. We vary the values of $K$ and $T$, which denote the
volumes of memory bank employed in the cross-domain/class distillation and
historical self-distillation modules. The choice of $\tau$ in Eq. (6) which
coordinates the identification of principle subspace is also studied. The
results indicate that the choices of those hyperparameters do not have a
significant influence as long as they are set within reasonable intervals.
Compared with them, the choices of $\lambda$ in Eq. (8) and $\kappa$ in Eq.
(11) demonstrate more importance and are required to be carefully decided.

###  4.6 Robustness to Support Sample Selection

To evaluate the robustness of our method against the randomness during few-
shot sample selection, we run the experiments for 10 times with different sets
of labeled samples in the target domain. The averaged results are presented in
Table 8. It demonstrates that our method consistently outperforms the
competing approaches under diverse settings, which indicates its strong
robustness.

## References

  * [1] Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang.  Generalization and equilibrium in generative adversarial nets (gans).  In International Conference on Machine Learning, pages 224–232. PMLR, 2017.
  * [2] Sheikh Shams Azam, Seyyedali Hosseinalipour, Qiang Qiu, and Christopher Brinton.  Recycling model updates in federated learning: Are gradient subspaces low-rank?  In International Conference on Learning Representations, 2022.
  * [3] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira.  Analysis of representations for domain adaptation.  Advances in neural information processing systems, 19, 2006.
  * [4] Debora Caldarola, Barbara Caputo, and Marco Ciccone.  Improving generalization in federated learning by seeking flat minima.  In European Conference on Computer Vision, pages 654–672. Springer, 2022.
  * [5] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park.  Swad: Domain generalization by seeking flat minima.  Advances in Neural Information Processing Systems, 34:22405–22418, 2021.
  * [6] Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina.  Entropy-SGD: Biasing gradient descent into wide valleys.  In International Conference on Learning Representations, 2017.
  * [7] Cheng Chen, Qi Dou, Hao Chen, Jing Qin, and Pheng-Ann Heng.  Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 865–872, 2019.
  * [8] Wentao Chen, Zhang Zhang, Wei Wang, Liang Wang, Zilei Wang, and Tieniu Tan.  Cross-domain cross-set few-shot learning via learning compact and aligned representations.  In European Conference on Computer Vision, pages 383–399. Springer, 2022.
  * [9] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich.  Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks.  In International conference on machine learning, pages 794–803. PMLR, 2018.
  * [10] Ali Cheraghian, Shafin Rahman, Sameera Ramasinghe, Pengfei Fang, Christian Simon, Lars Petersson, and Mehrtash Harandi.  Synthesized feature based few-shot class-incremental learning on a mixture of subspaces.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8661–8670, 2021.
  * [11] Joseph Paul Cohen.  Covid-19 image data collection: Prospective predictions are the future.  arXiv:2006.11988, 2020.
  * [12] Felix Dangel, Frederik Kunstner, and Philipp Hennig.  Backpack: Packing more into backprop.  In International Conference on Learning Representations, 2020.
  * [13] Debasmit Das, Sungrack Yun, and Fatih Porikli.  ConfeSS: A framework for single source cross-domain few-shot learning.  In International Conference on Learning Representations, 2022.
  * [14] Susanna Dodd, Mike Clarke, Lorne Becker, Chris Mavergames, Rebecca Fish, and Paula R Williamson.  A taxonomy has been developed for outcomes in medical research to help improve knowledge discovery.  Journal of clinical epidemiology, 96:84–92, 2018.
  * [15] C Domínguez Conde, C Xu, LB Jarvis, DB Rainbow, SB Wells, T Gomes, SK Howlett, O Suchanek, K Polanski, HW King, et al.  Cross-tissue immune cell analysis reveals tissue-specific features in humans.  Science, 376(6594), 2022.
  * [16] Zhekai Du, Jingjing Li, Hongzu Su, Lei Zhu, and Ke Lu.  Cross-domain gradient discrepancy minimization for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3937–3946, 2021.
  * [17] Jevgenij Gamper, Navid Alemi Koohbanani, Ksenija Benet, Ali Khuram, and Nasir Rajpoot.  Pannuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification.  In European congress on digital pathology, pages 11–19. Springer, 2019.
  * [18] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The journal of machine learning research, 17(1):2096–2030, 2016\.
  * [19] Zhiqiang Gao, Shufei Zhang, Kaizhu Huang, Qiufeng Wang, and Chaoliang Zhong.  Gradient distribution alignment certificates better adversarial domain adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8937–8946, 2021.
  * [20] Yixiao Ge, Dapeng Chen, and Hongsheng Li.  Mutual mean-teaching: Pseudo label refinery for unsupervised domain adaptation on person re-identification.  In International Conference on Learning Representations, 2020.
  * [21] Rui Gong, Martin Danelljan, Dengxin Dai, Danda Pani Paudel, Ajad Chhatkuli, Fisher Yu, and Luc Van Gool.  Tacs: Taxonomy adaptive cross-domain semantic segmentation.  In European Conference on Computer Vision, pages 19–35. Springer, 2022.
  * [22] Simon Graham, Mostafa Jahanifar, Ayesha Azam, Mohammed Nimir, Yee-Wah Tsang, Katherine Dodd, Emily Hero, Harvir Sahota, Atisha Tank, Ksenija Benes, et al.  Lizard: a large-scale dataset for colonic nuclear instance segmentation and classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 684–693, 2021.
  * [23] Simon Graham, Quoc Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, and Nasir Rajpoot.  Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images.  Medical Image Analysis, 58:101563, 2019.
  * [24] Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris.  A broader study of cross-domain few-shot learning.  In European conference on computer vision, pages 124–141. Springer, 2020.
  * [25] Sepp Hochreiter and Jürgen Schmidhuber.  Flat minima.  Neural computation, 9(1):1–42, 1997.
  * [26] Yanxu Hu and Andy J Ma.  Adversarial feature augmentation for cross-domain few-shot classification.  In European Conference on Computer Vision, pages 20–37. Springer, 2022.
  * [27] Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, and Richard J Radke.  Dynamic distillation network for cross-domain few-shot recognition with unlabeled data.  Advances in Neural Information Processing Systems, 34:3584–3595, 2021.
  * [28] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson.  Averaging weights leads to wider optima and better generalization.  Proceedings of the Conference on Uncertainty in Artificial Intelligence, 2018.
  * [29] Arthur Jacot, Franck Gabriel, and Clément Hongler.  Neural tangent kernel: Convergence and generalization in neural networks.  Advances in neural information processing systems, 31, 2018.
  * [30] Stanisław Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey.  Three factors influencing minima in sgd.  arXiv preprint arXiv:1711.04623, 2017.
  * [31] Sajid Javed, Arif Mahmood, Muhammad Moazam Fraz, Navid Alemi Koohbanani, Ksenija Benes, Yee-Wah Tsang, Katherine Hewitt, David Epstein, David Snead, and Nasir Rajpoot.  Cellular community detection for tissue phenotyping in colorectal cancer histology images.  Medical image analysis, 63:101696, 2020.
  * [32] Maggie Karthik.  Aptos 2019 blindness detection.  Kaggle.
  * [33] Jakob Nikolas Kather, Cleo-Aron Weis, Francesco Bianconi, Susanne M Melchers, Lothar R Schad, Timo Gaiser, Alexander Marx, and Frank Gerrit Zöllner.  Multi-class texture analysis in colorectal cancer histology.  Scientific reports, 6(1):1–11, 2016.
  * [34] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang.  On large-batch training for deep learning: Generalization gap and sharp minima.  In International Conference on Learning Representations, 2017.
  * [35] Jogendra Nath Kundu, Rahul Mysore Venkatesh, Naveen Venkat, Ambareesh Revanur, and R Venkatesh Babu.  Class-incremental domain adaptation.  In European Conference on Computer Vision, pages 53–69. Springer, 2020.
  * [36] Canran Li, Dongnan Liu, Haoran Li, Zheng Zhang, Guangming Lu, Xiaojun Chang, and Weidong Cai.  Domain adaptive nuclei instance segmentation and classification via category-aware feature alignment and pseudo-labelling.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 715–724. Springer, 2022.
  * [37] Tao Li.  Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening.  Information Sciences, 2019.
  * [38] Wei-Hong Li, Xialei Liu, and Hakan Bilen.  Cross-domain few-shot learning with task-specific adapters.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7161–7170, 2022.
  * [39] Xinyan Li, Qilong Gu, Yingxue Zhou, Tiancong Chen, and Arindam Banerjee.  Hessian based analysis of sgd for deep nets: Dynamics and generalization.  In Proceedings of the 2020 SIAM International Conference on Data Mining, pages 190–198. SIAM, 2020.
  * [40] Hanwen Liang, Qiong Zhang, Peng Dai, and Juwei Lu.  Boosting the generalization capability in cross-domain few-shot learning via noise-enhanced supervised autoencoder.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9424–9434, 2021.
  * [41] Nina Linder, Juho Konsti, Riku Turkki, Esa Rahtu, Mikael Lundin, Stig Nordling, Caj Haglund, Timo Ahonen, Matti Pietikäinen, and Johan Lundin.  Identification of tumor epithelium and stroma in tissue microarrays using texture analysis.  Diagnostic pathology, 7:1–11, 2012.
  * [42] Dongnan Liu, Chaoyi Zhang, Yang Song, Heng Huang, Chenyu Wang, Michael Barnett, and Weidong Cai.  Decompose to adapt: Cross-domain object detection via feature disentanglement.  IEEE Transactions on Multimedia, 25:1333–1344, 2022.
  * [43] Dongnan Liu, Donghao Zhang, Yang Song, Fan Zhang, Lauren O’Donnell, Heng Huang, Mei Chen, and Weidong Cai.  Unsupervised instance segmentation in microscopy images via panoptic domain adaptation and task re-weighting.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4243–4252, 2020.
  * [44] Xu Luo, Jing Xu, and Zenglin Xu.  Channel importance matters in few-shot image classification.  In International Conference on Machine Learning, pages 14542–14559. PMLR, 2022.
  * [45] Zelun Luo, Yuliang Zou, Judy Hoffman, and Li F Fei-Fei.  Label efficient learning of transferable representations acrosss domains and tasks.  Advances in neural information processing systems, 30, 2017.
  * [46] Deborah Lupton.  Medicine as culture: illness, disease and the body.  Sage, 2012.
  * [47] Lucas Mansilla, Rodrigo Echeveste, Diego H Milone, and Enzo Ferrante.  Domain generalization via gradient surgery.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6630–6638, 2021.
  * [48] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In 2016 fourth international conference on 3D vision (3DV), pages 565–571. Ieee, 2016.
  * [49] Pau Panareda Busto and Juergen Gall.  Open set domain adaptation.  In Proceedings of the IEEE international conference on computer vision, pages 754–763, 2017.
  * [50] Cheng Perng Phoo and Bharath Hariharan.  Self-training for few-shot transfer across extreme task differences.  In International Conference on Learning Representations, 2021.
  * [51] Pranav Raikote.  Covid-19 image dataset.  Kaggle.
  * [52] Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  In International Conference on Learning Representations, 2017.
  * [53] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko.  Semi-supervised domain adaptation via minimax entropy.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8050–8058, 2019.
  * [54] Ozan Sener and Vladlen Koltun.  Multi-task learning as multi-objective optimization.  Advances in neural information processing systems, 31, 2018.
  * [55] Yuge Shi, Jeffrey Seely, Philip Torr, Siddharth N, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve.  Gradient matching for domain generalization.  In International Conference on Learning Representations, 2022.
  * [56] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  Advances in neural information processing systems, 30, 2017.
  * [57] Antonio Torralba and Alexei A Efros.  Unbiased look at dataset bias.  In CVPR 2011, pages 1521–1528. IEEE, 2011.
  * [58] Philipp Tschandl, Christoph Rinner, Zoe Apalla, Giuseppe Argenziano, Noel Codella, Allan Halpern, Monika Janda, Aimilios Lallas, Caterina Longo, Josep Malvehy, et al.  Human–computer collaboration for skin cancer recognition.  Nature Medicine, 26(8):1229–1234, 2020.
  * [59] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler.  The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.  Scientific data, 5(1):1–9, 2018.
  * [60] Hung-Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang.  Cross-domain few-shot classification via learned feature-wise transformation.  In International Conference on Learning Representations, 2020.
  * [61] Hemanth Venkateswara.  Deep hashing network for unsupervised domain adaptation.  In CVPR, 2017.
  * [62] Qian Wang and Toby Breckon.  Unsupervised domain adaptation via structured prediction based selective pseudo-labeling.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 6243–6250, 2020.
  * [63] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni.  Generalizing from a few examples: A survey on few-shot learning.  ACM computing surveys (csur), 53(3):1–34, 2020.
  * [64] Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell.  Characterizing and avoiding negative transfer.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11293–11302, 2019.
  * [65] Julia K Winkler, Christine Fink, Ferdinand Toberer, Alexander Enk, Teresa Deinlein, Rainer Hofmann-Wellenhof, Luc Thomas, Aimilios Lallas, Andreas Blum, Wilhelm Stolz, et al.  Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition.  JAMA dermatology, 155(10):1135–1141, 2019.
  * [66] Mengya Xu, Mobarakol Islam, Chwee Ming Lim, and Hongliang Ren.  Class-incremental domain adaptation with smoothing and calibration for surgical report generation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 269–278. Springer, 2021.
  * [67] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan.  Universal domain adaptation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2720–2729, 2019.
  * [68] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn.  Gradient surgery for multi-task learning.  Advances in Neural Information Processing Systems, 33:5824–5836, 2020.
  * [69] Wang Yuan, Zhizhong Zhang, Cong Wang, Haichuan Song, Yuan Xie, and Lizhuang Ma.  Task-level self-supervision for cross-domain few-shot learning.  Proceedings of the AAAI Conference on Artificial Intelligence, 36(3):3215–3223, 2022.
  * [70] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023–6032, 2019.
  * [71] Qiming Zhang, Jing Zhang, Wei Liu, and Dacheng Tao.  Category anchor-guided unsupervised domain adaptation for semantic segmentation.  Advances in neural information processing systems, 32, 2019.
  * [72] Lin Zuo, Mengmeng Jing, Jingjing Li, Lei Zhu, Ke Lu, and Yang Yang.  Challenging tough samples in unsupervised domain adaptation.  Pattern Recognition, 110:107540, 2021.


IR: infrared
JWST: James Webb Space Telescope
L2: Lagrange point 2
LG: Local Group
LSST: Large Synoptic Survey Telescope
MPG: Max Planck Gesellschaft
MPs: multiple stellar populations
MS: main sequence
MW: Milky Way
NASA: National Aeronautics and Space Administration
PM: proper motion
PSF: point-spread function
QE: quantum efficiency
UVIS: Ultraviolet-VISible
VIRCAM: VISTA InfraRed CAMera
VIRGO: Raytheon HgCdTe 0.84-2.5 micron, 2048x2048 pixel IR detectors (see Ref.
104)
VISTA: Visible and Infrared Survey Telescope for Astronomy
VLT: Very Large Telescope
WFC: Wide-Field Channel
WFC3: Wide-Field Camera 3
WFI: Wide-Field Imager
WFIRST: Wide-Field InfraRed Space Telescope


h_{q}^{2}}=0.$

Periodic boundary conditions are imposed, _i.e._
$[\Psi^{h}_{0}]_{0,:}=[\Psi^{h}_{0}]_{m_{q},:}$ and
$[\Psi^{h}_{0}]_{:,0}=[\Psi^{h}_{0}]_{:,m_{q}}$.

##### Discretization of perturbations

We now consider several classes of perturbation operators. One typical form is
$\widetilde{\mathcal{L}}=F^{T}\nabla$, with $F=(F_{1},F_{2})\in\mathbb{R}^{2}$
for two-dimensional dynamics. Its $L^{2}$-adjoint acts as

(179)
$\widetilde{\mathcal{L}}^{\dagger}\psi_{0}=-\operatorname{div}(F)\psi_{0}-F^{T}\nabla\psi_{0}.$

Perturbations of this form include the physical perturbations
$\widetilde{\mathcal{L}}_{\mathrm{phys}}$ considered throughout this work,
divergence-free vector fields (116), and the differential term in the
Feynman–Kac forcing (124). In dimension one, the-right hand side of (179) is
discretized with a centered finite difference as

(180)
$-[F^{\prime}]_{i}[\Psi^{h}_{0}]_{i}-[F]_{i}\frac{[\Psi^{h}_{0}]_{i+1}-[\Psi^{h}_{0}]_{i-1}}{2h_{q}}.$

In dimension two, perturbations of the form (179) are discretized with
centered finite differences as

(181)
$\displaystyle-\left([F_{1}]_{i,j}\frac{[\Psi^{h}_{0}]_{i+1,j}-[\Psi^{h}_{0}]_{i-1,j}}{2h_{q}}+[F_{2}]_{i,j}\frac{[\Psi^{h}_{0}]_{i,j+1}-[\Psi^{h}_{0}]_{i,j-1}}{2h_{q}}\right)$
$\displaystyle-[\partial_{q_{1}}F_{1}]_{i,j}[\Psi^{h}_{0}]_{i,j}-[\partial_{q_{2}}F_{2}]_{i,j}[\Psi^{h}_{0}]_{i,j}.$

Another example is the modified fluctuation-dissipation perturbation (121),
preceeded by a factor $\alpha\eta$ as presented in this work. Since it
corresponds to the generator (28) of the dynamics at hand, its discretization
corresponds in any dimension to rescaling the discretized Fokker–Planck by
$(1+\alpha\eta)$.

We also consider zero order operators, such as the source term for the
Feynman–Kac forcing (124), namely $\xi^{T}\nabla V$. As terms of this form
include no differential operators acting on $\psi_{0}$, their discretization
is trivially done, in any dimension, by direct evaluation, _e.g._
$[V^{\prime}]_{i}[\Psi^{h}_{0}]_{i}$ in dimension one.

###  B.2 Langevin 1D

For the one-dimensional Langevin dynamics, we discretize the unbounded
momentum space as follows: we first truncate it to
$[-p_{\mathrm{max}},p_{\mathrm{max}}]$, then discretize it into $m_{p}$
interior points with uniform step size $h_{p}=2p_{\mathrm{max}}/(m_{p}-1)$.
For a continuous function $u\colon\mathbb{T}\times\mathbb{R}\to\mathbb{R}$, we
denote by $[u]_{i,j}=u(Q_{i},P_{j})$, where $Q_{i}=ih_{q}$ and $P_{j}=jh_{p}$
are the mesh points.

The numerical scheme for the Fokker–Planck equation (36) for Langevin dynamics
is obtained with centered finite differences, except for the transport term
$p^{T}M^{-1}\nabla_{q}\psi_{0}$, where an upwind scheme is used (see below for
details). In dimension one, (36) reads

(182)
$\mathcal{L}_{0}^{\dagger}\psi=-\mathcal{L}_{\mathrm{ham}}\psi+\gamma\left(M^{-1}\psi+M^{-1}p\partial_{p}\psi+\beta^{-1}\partial^{2}_{p}\psi\right).$

The discretization of (182) then reads

(183)
$\displaystyle[V^{\prime}]_{i}\frac{[\Psi^{h}_{0}]_{i,j+1}-[\Psi^{h}_{0}]_{i,j-1}}{2h_{p}}-\frac{P^{+}_{j}[\partial\Psi^{h}_{0}]_{i,j}^{-}+P^{-}_{j}[\partial\Psi^{h}_{0}]_{i,j}^{+}}{M}+\frac{\gamma
P_{j}}{M}\frac{[\Psi^{h}_{0}]_{i,j+1}-[\Psi^{h}_{0}]_{i,j-1}}{2h_{p}}$
$\displaystyle+\frac{\gamma}{M\beta}\frac{[\Psi^{h}_{0}]_{i,j+1}-2[\Psi^{h}_{0}]_{i,j}+[\Psi^{h}_{0}]_{i,j-1}}{h_{p}^{2}}+\frac{\gamma[\Psi^{h}_{0}]_{i,j}}{M}=0,$

where

(184) $p^{+}=\max(p,0),\qquad p^{-}=\min(p,0),$

and

(185)
$[\partial\Psi^{h}_{0}]_{i,j}^{+}=\frac{[\Psi^{h}_{0}]_{i+1,j}-[\Psi^{h}_{0}]_{i,j}}{h_{q}},\qquad[\partial\Psi^{h}_{0}]_{i,j}^{-}=\frac{[\Psi^{h}_{0}]_{i,j}-[\Psi^{h}_{0}]_{i-1,j}}{h_{q}}.$

Periodic boundary conditions are imposed on the positions, _i.e._
$[\Psi^{h}_{0}]_{0,j}=[\Psi^{h}_{0}]_{m_{q},j}$ for any $1\leqslant j\leqslant
m_{p}$.

##### Discretization of perturbations

For Langevin dynamics, perturbations of the form (179) can correspond to
differential operators acting on the positions or momenta, _i.e._
$F^{T}\nabla_{q}$ and $F^{T}\nabla_{p}$. In dimension one, these options
reduce to $-F\partial_{q}\psi_{0}$ and $-F\partial_{p}\psi_{0}$, both of which
are trivially discretized with centered finite differences.

For the modified fluctuation-dissipation in $p$, its discretization
corresponds to scaling the $\mathcal{L}_{\mathrm{FD}}$ term in (35) by
$(\gamma+\alpha\eta)$. For its position counterpart
$-\beta^{-1}\nabla^{*}_{q}\nabla_{q}$, its discretization corresponds to
adding (177), scaled by $\alpha\eta$.

##### Upwind scheme for Langevin

We now motivate the upwinding used to discretize the transport term
$p^{T}M^{-1}\nabla_{q}$ in the Fokker–Planck equation (36). In particular, the
use of a centered finite difference with an even number of points $m_{q}$ in
the spatial domain would lead to independent submeshes, and hence might give
incorrect results, an issue known as odd-even decoupling (see [45] for a
thorough discussion). To overcome this, one resorts to decentered finite
differences, in particular upwinding.

Recall that equation (7) can be seen as the stationary solution to the
evolution PDE $\partial_{t}\psi=\mathcal{L}^{\dagger}\psi$. To implement the
upwinding scheme, we write the transport term in the Fokker–Planck in the form
of the advection equation, that is

(186) $\partial_{t}\psi+p^{T}M^{-1}\nabla_{q}\psi=0.$

The scheme used to discretize the components of $\nabla_{q}\psi$ depends on
the sign of the components of $p$. The (partial) derivative is discretized
with a scheme decentered on the left when $p\geqslant 0$, and decentered on
the right when $p\leqslant 0$. The discretization of $p\partial_{q}\psi$ in
dimension one at a mesh point $(Q_{i},P_{j})$ is therefore done as follows:

(187)
$\begin{cases}p\dfrac{[\Psi^{h}_{0}]_{i+1,j}-[\Psi^{h}_{0}]_{i,j}}{h_{q}},\qquad
p<0,\\\ p\dfrac{[\Psi^{h}_{0}]_{i,j}-[\Psi^{h}_{0}]_{i-1,j}}{h_{q}},\qquad
p>0.\end{cases}$

## Acknowledgments

This project has received funding from the European Union’s Horizon 2020
research and innovation program under the Marie Sklodowska–Curie grant
agreement No 945332, and from the European Research Council (ERC) under the
European Union’s Horizon 2020 research and innovation programme (project EMC2,
grant agreement No 810367). We also acknowledge funding from the Agence
Nationale de la Recherche, under grants ANR-19-CE40-0010-01 (QuAMProcs) and
ANR-21-CE40-0006 (SINEQ).

## References

  * [1] M. P. Allen and D. J. Tildesley, Computer Simulation of Liquids, Oxford University Press, 2017, https://doi.org/10.1093/oso/9780198803195.001.0001.
  * [2] C. Barbarosie, Representation of divergence-free vector fields, Quarterly of Applied Mathematics, 69 (2011), pp. 309–316.
  * [3] G. Battimelli, G. Ciccotti, and P. Greco, Computer Meets Theoretical Physics: The New Frontier of Molecular Simulation, The Frontiers Collection, Springer, 2020, https://doi.org/10.1007/978-3-030-39399-1.
  * [4] R. Bhattacharya, On the functional central limit theorem and the law of the iterated logarithm for Markov processes, Probability Theory and Related Fields, 60 (1982), pp. 185–201, https://doi.org/10.1007/BF00531822.
  * [5] R. Caflisch, Monte Carlo and quasi-Monte Carlo methods, Acta Numerica, 7 (1998), pp. 1–49.
  * [6] G. Ciccotti, R. Kapral, and A. Sergi, Handbook of Materials Modeling: Methods, Springer, 2005, ch. Non-Equilibrium Molecular Dynamics, pp. 745–761.
  * [7] L. de Sousa Oliveira and A. Greaney, Method to manage integration error in the Green-Kubo method, Phys. Rev. E, 95 (2017), p. 023308, https://doi.org/10.1103/PhysRevE.95.023308.
  * [8] A. Dhar, Heat transport in low-dimensional systems, Advances in Physics, 57 (2008), pp. 457–537, https://doi.org/10.1080/00018730802538522.
  * [9] A. Duncan, N. Nuesken, and G. Pavliotis, Using perturbed underdamped Langevin dynamics to efficiently sample from probability distributions, Journal of Statistical Physics, 169 (2017), pp. 1098–1131, https://doi.org/10.1007/s10955-017-1906-8.
  * [10] A. B. Duncan, G. A. Pavliotis, and K. C. Zygalakis, Nonreversible Langevin samplers: Splitting schemes, analysis and implementation, 2017, https://arxiv.org/abs/1701.04247.
  * [11] D. Evans and G. Morriss, Statistical Mechanics of Nonequilibrium Liquids, ANU Press, 2007.
  * [12] M. Fathi, A. Homman, and G. Stoltz, Error analysis of the transport properties of Metropolized schemes, ESAIM: Proceedings and Surveys, 48 (2015), pp. 341–363.
  * [13] M. Fathi and G. Stoltz, Improving dynamical properties of metropolized discretizations of overdamped Langevin dynamics, Numerische Mathematik, 136 (2016), pp. 545–602, https://doi.org/10.1007/s00211-016-0849-3.
  * [14] P. Ferrari, S. Goldstein, and J. Lebowitz, Statistical Physics and Dynamical Systems: Rigorous Results, Birkhäuser Boston, 1985, ch. Diffusion, Mobility and the Einstein Relation, pp. 405–441.
  * [15] G. Ferré, M. Rousset, and G. Stoltz, More on the long time stability of Feynman–Kac semigroups, Stochastics and Partial Differential Equations: Analysis and Computations, 9 (2021), pp. 630–673, https://doi.org/10.1007/s40072-020-00178-3.
  * [16] G. Ferré and G. Stoltz, Error estimates on ergodic properties of Feynman–Kac semigroups, Numerische Mathematik, 143 (2019), pp. 261–313, https://doi.org/10.1007/s00211-019-01059-1.
  * [17] D. Frenkel and B. Smit, Understanding Molecular Simulation: From Algorithms to Applications, Academic Press, 2nd ed., 2002.
  * [18] M. J. Gillan and M. Dixon, The calculation of thermal conductivities by perturbed molecular dynamics simulation, Journal of Physics C: Solid State Physics, 16 (1983), pp. 869–878.
  * [19] E. M. Gosling, I. McDonald, and K. Singer, On the calculation by molecular dynamics of the shear viscosity of a simple fluid, Molecular Physics, 26 (1973), pp. 1475–1484, https://doi.org/10.1080/00268977300102631.
  * [20] M. S. Green, Markoff random processes and the statistical mechanics of time-dependent phenomena. II. Irreversible processes in fluids, The Journal of Chemical Physics, 22 (1954), pp. 398–413, https://doi.org/10.1063/1.1740082.
  * [21] M. Hairer and J. C. Mattingly, Yet another look at Harris’ ergodic theorem for Markov chains, in Seminar on Stochastic Analysis, Random Fields and Applications VI, vol. 63 of Progr. Probab., Birkhäuser/Springer, 2011, pp. 109–117.
  * [22] R. Khasminskii, Stochastic Stability of Differential Equations, vol. 66 of Stochastic modelling and applied probability, Springer, 2nd ed., 2012\.
  * [23] W. Kliemann, Recurrence and invariant measures for degenerate diffusions, The Annals of Probability, 15 (1987), pp. 690–707, https://doi.org/10.1214/aop/1176992166.
  * [24] M. Kopec, Weak backward error analysis for overdamped Langevin processes, IMA Journal of Numerical Analysis, 35 (2014), pp. 583–614, https://doi.org/10.1093/imanum/dru016.
  * [25] M. Kopec, Weak backward error analysis for Langevin process, BIT Numerical Mathematics, 55 (2015), pp. 1057–1103, https://doi.org/10.1007/s10543-015-0546-0.
  * [26] R. Kubo, Statistical-mechanical theory of irreversible processes. I. General theory and simple applications to magnetic and conduction problems, Journal of the Physical Society of Japan, 12 (1957), pp. 570–586, https://doi.org/10.1143/JPSJ.12.570.
  * [27] B. Leimkuhler and C. Matthews, Molecular Dynamics: With Deterministic and Stochastic Numerical Methods, Springer, 2015, https://doi.org/10.1007/978-3-319-16375-8.
  * [28] B. Leimkuhler, C. Matthews, and G. Stoltz, The computation of averages from equilibrium and nonequilibrium Langevin molecular dynamics, IMA Journal of Numerical Analysis, 36 (2016), pp. 13–79, https://doi.org/10.1093/imanum/dru056.
  * [29] T. Lelièvre and G. Stoltz, Partial differential equations and stochastic methods in molecular dynamics, Acta Numerica, 25 (2016), p. 681–880.
  * [30] T. Lelièvre, G. Stoltz, and M. Rousset, Free Energy Computations: A Mathematical Perspective, Imperial College Press, 2010, https://doi.org/10.1142/p579.
  * [31] S. Lepri, ed., Thermal Transport in Low Dimensions: From Statistical Physics to Nanoscale Heat Transfer, vol. 921, Springer International Publishing, 2016, https://doi.org/10.1007/978-3-319-29261-8.
  * [32] S. Lepri, R. Livi, and A. Politi, Thermal conduction in classical low-dimensional lattices, Physics Reports, 377 (2003), pp. 1–80, https://doi.org/10.1016/S0370-1573(02)00558-6.
  * [33] X. Mao, Stochastic Differential Equations and Applications, Woodhead Publishing, 2nd ed., 2008.
  * [34] J. Mattingly, A. Stuart, and D. Higham, Ergodicity for sdes and approximations: locally Lipschitz vector fields and degenerate noise, Stochastic Processes and their Applications, 101 (2002), pp. 185–232.
  * [35] P. D. Moral, Feynman–Kac Formulae, Springer New York, 2004, https://doi.org/10.1007/978-1-4684-9393-1.
  * [36] E. Nelson, Dynamical Theories of Brownian Motion, Princeton University Press, 1967.
  * [37] P. Plechac, G. Stoltz, and T. Wang, Convergence of the likelihood ratio method for linear response of non-equilibrium stationary states, ESAIM:M2AN, 55 (2021), pp. S593–S623, https://doi.org/10.1051/m2an/2020050.
  * [38] P. Plechac, G. Stoltz, and T. Wang, Martingale product estimators for sensitivity analysis in computational statistical physics, IMA Journal of Numerical Analysis, (2022), https://doi.org/10.1093/imanum/drac073.
  * [39] P. Reimann, Brownian motors: noisy transport far from equilibrium, Physics Reports, 361 (2002), pp. 57–265.
  * [40] L. Rey-Bellet, Ergodic properties of Markov processes, in Open Quantum Systems II, S. Attal, A. Joye, and C.-A. Pillet, eds., vol. 1881 of Lecture Notes in Mathematics, Springer, 2006, pp. 1–39, https://doi.org/10.1007/3-540-33966-3_1.
  * [41] L. Rey-Bellet and K. Spiliopoulos, Irreversible Langevin samplers and variance reduction: a large deviations approach, Nonlinearity, 28 (2015), pp. 2081–2103, https://doi.org/10.1088/0951-7715/28/7/2081.
  * [42] J. Roussel and G. Stoltz, A perturbative approach to control variates in molecular dynamics, Multiscale Modeling & Simulation, 17 (2019), pp. 552–591, https://doi.org/10.1137/18M1171047.
  * [43] G. Stoltz, Error estimates and variance reduction for nonequilibrium stochastic dynamics, 2022, https://arxiv.org/abs/2211.10717.
  * [44] D. Talay, Stochastic Hamiltonian systems: Exponential convergence to the invariant measure, and discretization by the implicit Euler scheme, Markov Processes and Related Fields, 8 (2002), pp. 163–198.
  * [45] H. Tang, W. Dong, and A. Agrawal, A phenomenon of artificial odd-even grid oscillation and its presence in domain decomposition computation: Algebraic analysis and numerical illustration, Journal of Computational and Applied Mathematics, 333 (2018), pp. 404–427, https://doi.org/10.1016/j.cam.2017.10.017.
  * [46] M. Tuckerman, Statistical Mechanics: Theory and Molecular Simulation, Oxford Graduate Texts, OUP Oxford, 2010.
  * [47] L. Wu, Large and moderate deviations and exponential convergence for stochastic damping Hamiltonian systems, Stochastic Processes and their Applications, 91 (2001), pp. 205–238, https://doi.org/10.1016/S0304-4149(00)00061-2.