11institutetext: Monash University, Melbourne, Victoria, Australia
22institutetext: UNC-Chapel Hill, Chapel Hill, NC, USA
22email: richard.peng.xia@gmail.com,huaxiu<EMAIL_ADDRESS>
# Generalizing to Unseen Domains in Diabetic Retinopathy with Disentangled
Representations

Peng Xia Equal Contribution1122 Ming Hu⋆ 11 Feilong Tang 11 Wenxue Li 11
Wenhao Zheng 22
Lie Ju 11 Peibo Duan 11 Huaxiu Yao 22 Zongyuan Ge 11

###### Abstract

Diabetic Retinopathy (DR), induced by diabetes, poses a significant risk of
visual impairment. Accurate and effective grading of DR aids in the treatment
of this condition. Yet existing models experience notable performance
degradation on unseen domains due to domain shifts. Previous methods address
this issue by simulating domain style through simple visual transformation and
mitigating domain noise via learning robust representations. However, domain
shifts encompass more than image styles. They overlook biases caused by
implicit factors such as ethnicity, age, and diagnostic criteria. In our work,
we propose a novel framework where representations of paired data from
different domains are decoupled into semantic features and domain noise. The
resulting augmented representation comprises original retinal semantics and
domain noise from other domains, aiming to generate enhanced representations
aligned with real-world clinical needs, incorporating rich information from
diverse domains. Subsequently, to improve the robustness of the decoupled
representations, class and domain prototypes are employed to interpolate the
disentangled representations while data-aware weights are designed to focus on
rare classes and domains. Finally, we devise a robust pixel-level semantic
alignment loss to align retinal semantics decoupled from features, maintaining
a balance between intra-class diversity and dense class features. Experimental
results on multiple benchmarks demonstrate the effectiveness of our method on
unseen domains. The code implementations are accessible on
https://github.com/richard-peng-xia/DECO.

###### Keywords:

Diabetic Retinopathy Domain Generalization Disentangled Representations.

##  1 Introduction

Diabetic Retinopathy (DR) is a diabetes-induced ocular disorder that affects
the retina, which epitomizes one of the foremost causes of blindness [25].
Typically, the diagnosis of DR is based on the presence of several key
lesions, namely microaneurysms, hemorrhages, soft or hard exudates,
hemorrhages, and cotton wool spots. Therefore, the grading of DR usually
includes five categories: no DR, mild DR, moderate DR, severe DR, and
proliferative DR [20].

Figure 1: (Left) An example of fundus-based domain variances. The horizontal
distance represents domain differences, and the vertical distance denotes DR
category differences. (Right) The motivations of our approach. Firstly, while
the augmentation methods are simple visual transformations, we consider more
feature-level class-agnostic latent noise, such as macular degeneration caused
by age. Additionally, existing pixel-level alignment may act on features
containing domain bias, replacing original features with decoupled semantic
features to alleviate domain noise.

Although deep learning methods have achieved promising results in grading DR
[4, 9, 14, 17], which simplifies the diagnostic process and reduces the demand
for trained ophthalmologists, one major challenge they face in practical
clinical applications is domain shifts [2, 3, 4, 6], meaning there are some
visual biases between training and testing data caused by factors such as
imaging conditions or the ethnic of the population, as shown in Figure 1. This
leads to a decline in models’ performance when these models are applied to new
data from unseen domains, which is known as Domain Generalization (DG) [29,
32, 35, 24].

Previous efforts have sought to learn robust domain-invariant representations,
such as through flatness and regularization [2, 33], and variational
autoencoders [6]. There are works from the perspective of image augmentation
[3, 32, 31], using visual transformation or image degradation to simulate
domain shifts. However, generalized features are likely affected by domain
biases unrelated to DR categories, such as macular degeneration in elderly
retinas and ethnic variations in retinal structure. This renders current data
augmentation schemes and direct feature learning ineffective when confronted
with new unseen domains. Additionally, there are works proposing pixel-level
supervised losses to learn diverse intra-class features [3], but the targets
themselves contain domain noise, which makes the representations less robust.

Considering these drawbacks, we propose to Decouple the rEpresentations of
semantiC features from dOmain features to reduce domain bias, which we call
DECO. Specifically, features are decoupled into representations of semantic
features (DR-related retinal semantics, e.g., microaneurysms, hemorrhages,
hard/soft exudates) and domain features (domain noise, e.g., explicit noise in
image style and implicit noise stemming from age, gender, and ethnicity,
etc.). To mitigate the impact of domain shifts, we average the domain
information over examples of the same class and construct class prototype
representations. Then we linearly interpolate each semantic representation
using the corresponding class prototype. Similarly, we interpolate domain
representations with class-agnostic domain factors to improve training
stability and remove noise. During the insertion process, we design data-aware
weights to focus on rare classes and domains. By utilizing a set of data from
different domains, new representations containing the semantic features of one
example and domain features from another can be reassembled for data
augmentation. This approach can effectively improve the model’s generalization
ability and allows for more targeted data sampling from the perspective of
domain or class, especially for rare classes or domains, thus achieving
overall performance stability. In Domain Generalization (DG), sufficient
intra-class variability is crucial, a functionality not achievable through
traditional image-level supervised losses. Therefore, in DR grading, the
diverse diagnostic patterns across different domains encourage the model to
learn retinal lesion semantics at the pixel level as extensively as possible,
as pixel-level supervised representations are likely to be noisy, which is
overlooked in prior works [3]. By introducing decoupled retinal semantics for
pixel-level alignment alongside severity supervision at the image level, we
encourage the model to learn features with adequate intra-class variability
while preserving diagnostic patterns.

Our contributions can be summarized as: (1) We propose decoupling the
representation of semantic features from domain features and utilizing a
combination of semantic features and features from different domains for
feature-level data augmentation. Moreover, to improve the robustness of the
disentangled representations, class and domain prototypes are separately
interpolated into their corresponding representations; (2) We design a robust
pixel-level alignment loss to align retinal semantics decoupled from
representations; (3) Extensive experiments on comprehensive benchmarks
demonstrate the effectiveness of our framework on unseen domains.

Figure 2: The overview of our proposed method. (a) Representation decoupling
and recombination. (b) Representation enhancement with class and domain
prototypes. The specific process is shown in the right panel. (c) Robust
pixel-level semantic alignment.

##  2 Methodology

To achieve class-unbiased and domain-invariant representations, we propose DE-
CO, outlined as shown in Figure 2. The key concept is to decouple retinal
image representation into class-related retinal semantics and domain noise.
DECO migrates domain noise to all instances, which offers greater adaptability
compared to manually designed image augmentation methods [21, 15].
Additionally, DECO averages the semantic representations of each category and
domain noise separately to interpolate the representation of each instance.
Finally, we extract retinal semantics from the features for further pixel-
level alignment, enabling the model to learn robust class-related retinal
features.

Task Settings. In DG, let $\mathcal{D}=\\{D_{1},\ldots,D_{S}\\}$ denote the
source domains, where each domain $D_{i}$ includes data triplets
$(x_{i},y_{i},d)$ representing input fundus image $x_{i}$, label $y_{i}$, and
the domain $d$, drawn from the domain-specific distribution $P_{i}(x,y)$. For
each domain $d$, we define the number of training examples in each class as
$n_{i}=\\{n_{d,1},\ldots,n_{d,C}\\}$. The goal of DG is to learn a function
$f:\mathcal{X}\rightarrow\mathcal{Y}$ that minimizes the expected loss over an
unseen target domain $D_{T}$ with its own unique distribution $P_{T}(x,y)$.

Decoupling and Recombination of Representations. DECO constructs augmented
instances by combining the semantic representation of one instance with the
domain noise of another, thereby recombining pairs of instances to augment
instances of diverse styles. Inspired by style transfer [12], we employ
instance normalization [22] to decouple semantic and domain noise. In an
exemplar $(x,y,d)$, the hidden representation at the $l$-th ($l\textless L$)
layer is denoted as $r=f^{l}(x)\in\mathbb{R}^{C\times H\times W}$, where $C$,
$H$, and $W$ signify the channel, height, and width dimensions, respectively.
InstanceNorm normalizes it as:

$z(r)=\text{InstanceNorm}(r)=\frac{r-\mu(r)}{\sigma(r)}.$ (1)

where $\mu(\cdot)$, $\sigma(\cdot)$ are the mean and standard deviation
computed across the spatial dimensions of $r$ for each channel and each
sample:

$\small\mu(r)=\frac{1}{HW}\sum_{h=1}^{H}\sum_{w=1}^{W}r_{h,w},\;\;\sigma(r)=\sqrt{\frac{1}{HW}\sum_{h=1}^{H}\sum_{w=1}^{W}(r_{h,w}-\mu(r))^{2}}.$
(2)

InstanceNorm occurs in feature space, where the affine parameters can alter
the style of the image [12, 22]. Therefore, we consider the normalized example
$z(r)$ as the semantic representation, representing the retinal-related
semantics in retinal images, while $\mu(r)$ and $\sigma(r)$ are regarded as
domain noise, encompassing factors such as image style, background, ethnicity,
and age differences.
After decoupling representations, an augmented representation is generated by
interchanging semantic representations and domain noise within a pair of
examples $(x_{i},y_{i},d_{i})$ and $(x_{j},y_{j},d_{j})$.

$r^{\prime}=\sigma({r}_{i})\left(\frac{r_{i}-\mu(r_{i})}{\sigma(r_{i})}\right)+\mu(r_{j}),\;\;\hat{y}=y_{i}.$
(3)

Combining the semantic representation of $x_{i}$ with the domain
representation of $x_{j}$ yields an augmented representation $r^{\prime}$ with
the category $y_{i}$. By decoupling and recombining representations, numerous
augmented representations can be generated, encompassing diverse domain noise,
particularly making DECO more reliable when faced with domain or class
imbalance.

Prototype-enhanced Synthetic Representations. Although instance normalization
can effectively distinguish semantic information from domain noise, it is
challenging to entirely shield the semantic information of each instance from
domain bias. Therefore, to improve the robustness of semantic representations,
we leverage the averaged semantic representations from the same class but
different domains [27]. Considering the diversity of the same class, we
balance class and domain invariance by combining corresponding class prototype
representations and semantic representations. The class prototype
representation $p_{c}$ is an invariant representation belonging to class $c$,
independent of the domain, obtained by averaging semantic representations of
instances belonging to class $c$:

$p_{c}=\frac{1}{n_{\star,c}}\sum_{i=1}^{n_{\star,c}}z(r_{i})=\frac{1}{n_{\star,c}}\sum_{i=1}^{n_{\star,c}}\frac{r_{i}-\mu(r_{i})}{\sigma(r_{i})},\;\;\text{where}\;\;n_{\star,c}=\sum_{k=1}^{S}n_{k,c}.\\\
$ (4)

For every instance $(x,y,d)$ where $y=c$, the prototype-enhanced semantic
representation $\hat{z}(r)$ is derived through the interpolation of $z(r)$
with the class prototype $p_{c}$. Additionally, considering the class
imbalance [26, 11], we design a class-aware weight with the interpolation
coefficient $\lambda_{c}$ to balance the semantic representations and
prototypes of minority groups.

$\displaystyle\hat{z}(r)=\lambda_{c}w_{c}z(r)+(1-\lambda_{c}w_{c})p_{c},\;\;\small{w_{c}=\frac{\sum_{c=1}^{C}\sum_{k=1}^{S}\left(n_{k,c}\right)^{\gamma_{c}}}{\sum_{k=1}^{S}\left(n_{k,c}\right)^{\gamma_{c}}}},$
(5)

where $\gamma_{c}$ is a scale hyper-parameter to provide more flexibility.
Similarly, in the ideal scenario, $\mu(r)$ and $\sigma(r)$ exclusively contain
domain noise representations, yet they may also include class-related
representations, such as certain subtle lesions. In this case, we obtain
domain prototype representations $u_{d}$ and $v_{d}$ by averaging examples
from different classes within the same domain to alleviate semantic
information:

$u_{d}=\frac{1}{n_{d,\star}}\sum_{i=1}^{n_{d,\star}}\mu(r_{i}),\quad
v_{d}=\frac{1}{n_{d,\star}}\sum_{i=1}^{n_{d,\star}}\sigma(r_{i}),\;\;\text{where}\;\;n_{d,\star}=\sum_{k=1}^{C}n_{d,k}.\\\
$ (6)

Then, we design a weighting scheme tailored to domains with inadequate
representation, followed by interpolating the domain noise of each instance
with domain prototypes using domain weights and a balancing coefficient:

$\centering\hat{\mu}(r)=\lambda_{d}w_{d}\mu(r)+(1-\lambda_{d}w_{d})u_{d},\quad\hat{\sigma}(r)=\lambda_{d}w_{d}\sigma(r)+(1-\lambda_{d}w_{d})v_{d},\@add@centering$
(7)

where
$w_{d}=\frac{\sum_{d=1}^{S}\sum_{k=1}^{C}\left(n_{d,k}\right)^{\gamma_{d}}}{\sum_{k=1}^{C}\left(n_{d,k}\right)^{\gamma_{d}}}$.
Finally, by replacing the original semantic representation and domain noise,
the prototype-enhanced augmented representation:

$\hat{r}^{\prime}=\hat{\sigma}({r}_{j})\hat{z}(r_{i})+\hat{\mu}(r_{j}),\;\;\hat{y}^{\prime}=y_{i}.$
(8)

Robust Pixel-level Semantic Alignment Loss. Image-level supervised losses
(such as cross-entropy) have been the most common, capable of learning dense
class features. However, sufficient intra-class variation is crucial for
domain generalization, with some works [3, 5] proposing to encourage models to
learn pixel-level supervision. However, they overlook that the representations
for alignment are susceptible to domain interference. Therefore, we introduce
decoupled retinal semantics for pixel-level alignment, along with image-level
severity supervision, to encourage the model to learn robust class features on
the retina while possessing features with sufficient intra-class variation.
For each instance $(x,y,d)$, the original features are $r=f^{L}(x)$, and its
augmented features are $\hat{r}^{\prime}$. Our training objective combines
$\mathcal{L}_{img}$ and $\mathcal{L}_{pixel}$ as follows:

$\footnotesize\mathcal{L}_{total}=\mathcal{L}_{CE}\left(\text{MLP}(\hat{r}^{\prime}),y\right)-\alpha\log\frac{\text{exp}(z(\hat{r}^{\prime})\cdot
z(r)/\tau)}{\sum_{k=1}^{2M}\mathds{1}_{[k\neq
i]}\text{exp}(z(\hat{r}^{\prime})\cdot z(r_{k})/\tau)},$ (9)

where $z(\cdot)$ represents semantic features, $\mathds{1}[k\neq
i]\in\\{0,1\\}$ is an indicator function that equals 1 when $k\neq i$, $\tau$
denotes the temperature parameter, $\alpha$ dynamically controls the task
weight and $M$ is the size of the randomly sampled mini-batch.

Table 1: Comparison with state-of-the-art approaches under the DG test.

Target | APTOS | DeepDR | FGADR | IDRID | Messidor | RLDR | Average
---|---|---|---|---|---|---|---
Metrics | AUC | ACC | F1 | AUC | ACC | F1 | AUC | ACC | F1 | AUC | ACC | F1 | AUC | ACC | F1 | AUC | ACC | F1 | AUC | ACC | F1
ERM | 75.0 | 44.4 | 38.9 | 77.0 | 39.5 | 34.3 | 66.2 | 32.0 | 27.1 | 82.3 | 50.0 | 44.1 | 79.1 | 60.7 | 43.4 | 75.9 | 36.5 | 35.7 | 75.9 | 43.8 | 37.3
Mixup [34] | 75.3 | 62.6 | 43.2 | 75.3 | 29.0 | 25.2 | 66.7 | 42.3 | 32.3 | 78.8 | 39.0 | 27.6 | 76.7 | 54.7 | 32.6 | 76.9 | 43.6 | 37.7 | 75.0 | 45.2 | 33.1
MixStyle [37] | 79.0 | 65.8 | 39.9 | 76.9 | 32.9 | 27.9 | 71.2 | 35.8 | 22.7 | 83.0 | 51.4 | 39.2 | 75.2 | 62.2 | 36.5 | 75.5 | 41.1 | 31.4 | 76.8 | 48.2 | 32.9
GREEN [17] | 75.1 | 53.8 | 38.9 | 76.4 | 28.1 | 24.9 | 69.5 | 41.3 | 31.5 | 79.9 | 41.3 | 32.2 | 75.8 | 52.0 | 36.8 | 74.8 | 34.0 | 34.4 | 75.3 | 41.8 | 33.1
CABNet [9] | 75.8 | 55.5 | 39.4 | 75.2 | 42.7 | 31.8 | 73.2 | 43.7 | 34.8 | 79.2 | 44.8 | 37.3 | 74.2 | 56.1 | 34.1 | 75.8 | 37.0 | 35.6 | 75.6 | 46.6 | 35.5
DDAIG [36] | 78.0 | 67.1 | 41.0 | 75.6 | 37.6 | 32.2 | 73.6 | 42.0 | 33.8 | 82.1 | 37.4 | 27.0 | 76.6 | 58.4 | 35.3 | 75.6 | 36.1 | 27.7 | 76.9 | 46.4 | 32.8
ATS [28] | 77.1 | 56.9 | 38.3 | 79.4 | 36.1 | 31.6 | 74.7 | 46.7 | 33.4 | 83.0 | 41.5 | 34.9 | 77.2 | 64.7 | 35.8 | 76.5 | 37.4 | 34.9 | 78.0 | 47.2 | 34.8
Fishr [19] | 79.2 | 66.6 | 43.4 | 81.1 | 48.1 | 34.4 | 73.3 | 44.4 | 34.4 | 82.7 | 40.3 | 27.6 | 76.4 | 65.1 | 41.1 | 77.4 | 36.8 | 34.7 | 78.4 | 50.2 | 35.9
MDLT [30] | 77.3 | 57.2 | 41.5 | 80.0 | 39.5 | 36.2 | 74.1 | 45.7 | 29.0 | 81.5 | 44.2 | 35.4 | 75.4 | 58.9 | 36.9 | 75.7 | 37.6 | 35.0 | 77.3 | 47.2 | 35.7
DRGen [2] | 79.9 | 58.1 | 40.2 | 83.0 | 38.7 | 34.1 | 69.4 | 41.7 | 24.7 | 84.7 | 44.6 | 37.4 | 79.0 | 60.1 | 40.5 | 79.5 | 43.1 | 37.0 | 79.3 | 47.7 | 37.3
VAE-DG [6] | 79.5 | 66.7 | 45.5 | 82.9 | 38.1 | 34.3 | 75.8 | 44.8 | 36.4 | 83.6 | 41.7 | 35.6 | 78.7 | 59.5 | 40.2 | 78.8 | 41.3 | 35.9 | 79.9 | 48.7 | 38.0
GDRNet [3] | 79.9 | 66.8 | 46.0 | 84.7 | 53.1 | 45.3 | 80.8 | 45.3 | 39.4 | 84.0 | 40.3 | 35.9 | 83.2 | 63.4 | 50.9 | 82.9 | 45.8 | 43.5 | 82.6 | 52.5 | 43.5
DECO(Ours) | 81.0 | 72.4 | 56.6 | 86.3 | 57.3 | 52.6 | 81.3 | 49.1 | 43.5 | 87.0 | 44.6 | 37.4 | 82.9 | 65.1 | 52.7 | 83.5 | 48.0 | 46.3 | 83.7 | 56.1 | 48.2

##  3 Experiments

Datasets and Metrics. We conduct a comprehensive evaluation of our method on
GDRBench [3], which involves two generalization ability evaluation settings
and eight popular public datasets. First, GDRBench adopts the classic leave-
one-domain-out protocol (DG test), which requires leaving one domain for
evaluation and training models on the rest. It involves six datasets,
including DeepDR [16], Messidor [1], IDRID [18], APTOS [13], FGADR [38], and
RLDR [23]. Additionally, it incorporates an extreme single-domain
generalization setup (ESDG test), following the train-on-single-domain
protocol on the above datasets for training, but with two extra large-scale
datasets, DDR [14] and EyePACS [8] for evaluation.

For evaluation, we report three critical metrics, namely accuracy (ACC), the
area under the ROC curve (AUC), and macro F1-score (F1).
Implementation Details. We used ResNet-50 [10] pre-trained on ImageNet [7] as
the backbone and a fully connected layer as the linear classifier. We use the
AdamW optimizer with a learning rate of $5\times 10^{-4}$, weight decay of
$10^{-4}$ and a batch size of 32. The model is trained for 100 epochs. In our
experiments, we apply cross-validation to tune all hyperparameters with grid
search.

Baselines. Following [3], our comparative algorithms, besides the naive
Empirical Risk Minimization (ERM), are mainly divided into three categories:
conventional DR classification algorithms [9, 17], domain generalization
techniques [2, 34, 36, 37, 28, 6], and feature representation methods [30,
19].

Table 2: Comparison with state-of-the-art approaches under the ESDG test.

Source | APTOS | DeepDR | FGADR | IDRID | Messidor | RLDR | Average
---|---|---|---|---|---|---|---
Metrics | AUC | ACC | F1 | AUC | ACC | F1 | AUC | ACC | F1 | AUC | ACC | F1 | AUC | ACC | F1 | AUC | ACC | F1 | AUC | ACC | F1
ERM | 66.4 | 53.2 | 31.6 | 70.7 | 47.3 | 31.2 | 55.3 | 5.6 | 7.1 | 69.6 | 56.5 | 33.9 | 70.6 | 51.3 | 33.7 | 70.1 | 27.3 | 26.4 | 67.1 | 40.2 | 27.3
Mixup [34] | 65.5 | 49.4 | 30.2 | 70.7 | 49.7 | 33.3 | 58.8 | 5.8 | 7.4 | 70.2 | 64.0 | 32.6 | 71.5 | 63.0 | 32.6 | 72.9 | 27.7 | 27.0 | 68.3 | 43.3 | 27.2
MixStyle [37] | 62.0 | 48.8 | 25.0 | 53.3 | 32.0 | 14.6 | 51.0 | 7.0 | 7.9 | 53.0 | 53.5 | 19.4 | 51.4 | 57.6 | 16.8 | 53.5 | 18.3 | 6.4 | 54.0 | 36.2 | 15.0
GREEN [17] | 67.5 | 52.6 | 33.3 | 71.2 | 44.6 | 31.1 | 58.1 | 5.7 | 6.9 | 68.5 | 60.7 | 33.0 | 71.3 | 54.5 | 33.1 | 71.0 | 31.9 | 27.8 | 67.9 | 41.7 | 27.5
CABNet [9] | 67.3 | 52.2 | 30.8 | 70.0 | 55.4 | 32.0 | 57.1 | 6.1 | 7.5 | 67.4 | 62.7 | 31.7 | 72.3 | 63.8 | 35.3 | 75.2 | 23.0 | 25.4 | 68.2 | 43.8 | 27.2
DDAIG [36] | 67.4 | 48.7 | 31.6 | 73.2 | 38.5 | 29.7 | 59.9 | 5.0 | 5.5 | 70.2 | 60.2 | 33.4 | 73.5 | 69.1 | 35.6 | 74.4 | 25.4 | 23.5 | 69.8 | 41.2 | 26.7
ATS [28] | 68.8 | 51.7 | 32.4 | 72.7 | 52.4 | 33.5 | 60.3 | 5.3 | 5.7 | 69.1 | 66.6 | 30.6 | 73.4 | 64.8 | 32.4 | 75.0 | 24.2 | 23.9 | 69.9 | 44.2 | 26.4
Fishr [19] | 64.5 | 61.7 | 31.0 | 72.1 | 61.0 | 30.1 | 56.3 | 6.0 | 7.2 | 71.8 | 48.0 | 30.6 | 74.3 | 52.0 | 33.8 | 78.6 | 19.3 | 21.3 | 69.6 | 41.3 | 25.7
MDLT [30] | 67.6 | 53.3 | 32.4 | 73.1 | 50.2 | 33.7 | 57.1 | 7.1 | 7.8 | 71.9 | 61.7 | 32.4 | 73.4 | 58.9 | 34.1 | 76.6 | 29.0 | 30.0 | 70.0 | 43.4 | 28.4
DRGen [2] | 69.4 | 60.7 | 35.7 | 78.5 | 39.4 | 31.6 | 59.8 | 6.8 | 8.4 | 70.8 | 67.7 | 30.6 | 77.0 | 64.5 | 37.4 | 78.9 | 19.0 | 21.2 | 72.4 | 43.0 | 27.5
VAE-DG [6] | 67.7 | 51.9 | 32.3 | 75.4 | 53.6 | 33.4 | 60.2 | 5.1 | 5.6 | 72.4 | 63.0 | 32.3 | 75.8 | 59.9 | 35.7 | 78.9 | 23.6 | 24.0 | 71.7 | 42.9 | 27.2
GDRNet [3] | 69.8 | 52.8 | 35.2 | 76.1 | 40.0 | 35.0 | 63.7 | 7.5 | 9.2 | 72.9 | 70.0 | 35.1 | 78.1 | 65.7 | 40.5 | 79.7 | 44.3 | 37.9 | 73.4 | 46.7 | 32.2
DECO(Ours) | 70.6 | 59.7 | 36.4 | 78.2 | 40.3 | 35.8 | 65.5 | 9.9 | 11.7 | 74.2 | 74.8 | 38.7 | 79.8 | 70.1 | 44.9 | 81.1 | 49.3 | 40.8 | 75.0 | 50.7 | 34.7

Comparison with SoTA methods under the DG test. Table 1 demonstrates that our
approach consistently outperforms previous state-of-the-art methods across all
datasets. Furthermore, notable improvements are observed, particularly in
domains with limited representation such as IDRiD, indicating the efficacy of
our method in clinical applications. The decoupled feature enhancement and
semantic alignment exhibit outstanding performance in scenarios with limited
samples. Traditional DR classification methods, which do not account for
domain shifts, exhibit weak generalization. In contrast, domain generalization
and feature representation methods show improvements over the baseline,
attributed to the strategic design addressing domain shifts. In summary, DECO
significantly outperforms these SoTA methods. The decoupling representations
effectively achieve feature-level data augmentation, while the incorporation
of class or domain prototypes further enhances the robustness of augmented
data. Additionally, robust pixel-level semantic alignment enables the model to
learn precise intra-class variations, thereby improving model generalization.

Generalization from a single source domain. We further comprehensively
evaluate the generalization performance by training on single-domain datasets
and testing on large-scale unseen datasets. The results, as shown in Table 2,
indicate a significant performance decrease for all methods, highlighting the
challenge of the task. Despite the decreased utilization of feature-level data
augmentation in the ESDG test, DECO still outperforms other methods on most
datasets due to its enhanced diversity within the same class samples and the
learned robust domain-invariant representations.

Ablation study on the components. In Table 3, we conduct an ablation analysis
on the three components. Firstly, the Augmented Disentangled Representations
(ADR) significantly improve the model’s generalization ability. Subsequently,
under the joint influence of class and domain prototypes, more robust
augmented representations further optimize the model. Finally, robust pixel-
level semantic alignment (RSALoss) of the disentangled semantic
representations enhances the model’s generalization by improving robust intra-
class diversity.

Further analysis of augmented methods and $\mathcal{L}_{pixel}$. To further
analyze the effectiveness of ADR and RSALoss, the primary components of our
approach, we compared them with state-of-the-art techniques in the same
category. For the augmentation methods, the comparative methods included the
standard DG augmentation [30], the default augmentation strategy in DRGen [2],
visual transformation and image degradation [3]. Our proposed disentangled
representation augmentation significantly outperforms the other methods.
Regarding semantic alignment loss, we compared models without semantic
alignment loss and with DahLoss [3]. Due to the improved robustness of
alignment features with decoupled semantic representations, RSALoss
demonstrates significant advantages.

Table 3: Ablation studies on proposed components under the DG test. ADR | Prototype | $\mathcal{L}_{pixel}$ | APTOS | DeepDR | FGADR | IDRID | Messidor | RLDR | Average
---|---|---|---|---|---|---|---|---|---
- | - | - | 75.26 | 77.21 | 66.35 | 82.44 | 79.29 | 75.95 | 76.08
✓ | - | - | 78.91 | 84.20 | 78.17 | 84.33 | 80.95 | 81.56 | 81.35
- | - | ✓ | 79.53 | 83.16 | 76.99 | 84.05 | 81.02 | 81.64 | 81.07
✓ | ✓ | - | 80.15 | 85.13 | 79.56 | 86.37 | 82.11 | 82.84 | 82.69
✓ | - | ✓ | 80.28 | 85.40 | 79.83 | 86.30 | 82.43 | 82.96 | 82.87
✓ | ✓ | ✓ | 81.03 | 86.29 | 81.26 | 86.99 | 82.94 | 83.50 | 83.67
Figure 3: Analysis of augmentation methods and $\mathcal{L}_{pixel}$ under the
DG test.

##  4 Conclusion

In this work, we mitigate the performance gap faced by DR grading models due
to domain shifts with disentangled representations. We innovatively propose a
method to combine semantic representations relevant to DR classes with domain
noise from another domain. It enhances the model’s generalization to unseen
domains. Then, to improve the robustness of decoupled representations, we
derive prototypes by averaging class and domain representations and designing
data-aware coefficients to adjust focus on different classes/domains,
facilitating interpolation between semantic representations and prototypes.
Finally, we introduce a robust pixel-level semantic alignment loss to
simultaneously learn densely packed class features while preserving inter-
class diversity.

## References

  * [1] Abràmoff, M.D., Lou, Y., Erginay, A., Clarida, W., Amelon, R., Folk, J.C., Niemeijer, M.: Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Investigative ophthalmology & visual science 57(13), 5200–5206 (2016)
  * [2] Atwany, M., Yaqub, M.: Drgen: Domain generalization in diabetic retinopathy classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 635–644. Springer (2022)
  * [3] Che, H., Cheng, Y., Jin, H., Chen, H.: Towards generalizable diabetic retinopathy grading in unseen domains. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 430–440. Springer (2023)
  * [4] Che, H., Jin, H., Chen, H.: Learning robust representation for joint grading of ophthalmic diseases via adaptive curriculum and feature disentanglement. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 523–533. Springer (2022)
  * [5] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)
  * [6] Chokuwa, S., Khan, M.H.: Generalizing across domains in diabetic retinopathy via variational autoencoders. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 265–274. Springer (2023)
  * [7] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [8] Emma, D., Jared, J., Will, C.: Eyepacs: Diabetic retinopathy detection (2015), https://www.kaggle.com/competitions/diabetic-retinopathy-detection
  * [9] He, A., Li, T., Li, N., Wang, K., Fu, H.: Cabnet: Category attention block for imbalanced diabetic retinopathy grading. IEEE Transactions on Medical Imaging 40(1), 143–153 (2020)
  * [10] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [11] Hu, M., Wang, L., Yan, S., Ma, D., Ren, Q., Xia, P., Feng, W., Duan, P., Ju, L., Ge, Z.: Nurvid: A large expert-level video database for nursing procedure activity understanding. In: Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2023)
  * [12] Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE international conference on computer vision. pp. 1501–1510 (2017)
  * [13] Karthick, M., Sohier, D.: Aptos 2019 blindness detection (2019), https://kaggle.com/competitions/aptos2019-blindness-detection
  * [14] Li, T., Gao, Y., Wang, K., Guo, S., Liu, H., Kang, H.: Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening. Information Sciences 501, 511–522 (2019)
  * [15] Liu, H., Li, H., Wang, X., Li, H., Ou, M., Hao, L., Hu, Y., Liu, J.: Understanding how fundus image quality degradation affects cnn-based diagnosis. In: 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). pp. 438–442. IEEE (2022)
  * [16] Liu, R., Wang, X., Wu, Q., Dai, L., Fang, X., Yan, T., Son, J., Tang, S., Li, J., Gao, Z., et al.: Deepdrid: Diabetic retinopathy—grading and image quality estimation challenge. Patterns 3(6) (2022)
  * [17] Liu, S., Gong, L., Ma, K., Zheng, Y.: Green: a graph residual re-ranking network for grading diabetic retinopathy. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part V 23. pp. 585–594. Springer (2020)
  * [18] Porwal, P., Pachade, S., Kamble, R., Kokare, M., Deshmukh, G., Sahasrabuddhe, V., Meriaudeau, F.: Indian diabetic retinopathy image dataset (idrid): a database for diabetic retinopathy screening research. Data 3(3), 25 (2018)
  * [19] Rame, A., Dancette, C., Cord, M.: Fishr: Invariant gradient variances for out-of-distribution generalization. In: International Conference on Machine Learning. pp. 18347–18377. PMLR (2022)
  * [20] Sebastian, A., Elharrouss, O., Al-Maadeed, S., Almaadeed, N.: A survey on deep-learning-based diabetic retinopathy classification. Diagnostics 13(3), 345 (2023)
  * [21] Shen, Z., Fu, H., Shen, J., Shao, L.: Modeling and enhancing low-quality retinal fundus images. IEEE transactions on medical imaging 40(3), 996–1006 (2020)
  * [22] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)
  * [23] Wei, Q., Li, X., Yu, W., Zhang, X., Zhang, Y., Hu, B., Mo, B., Gong, D., Chen, N., Ding, D., et al.: Learn to segment retinal lesions and beyond. In: 2020 25th International Conference on Pattern Recognition (ICPR). pp. 7403–7410. IEEE (2021)
  * [24] Wu, Z., Yao, H., Liebovitz, D., Sun, J.: An iterative self-learning framework for medical domain generalization. Advances in Neural Information Processing Systems 36 (2024)
  * [25] Wykoff, C.C., Khurana, R.N., Nguyen, Q.D., Kelly, S.P., Lum, F., Hall, R., Abbass, I.M., Abolian, A.M., Stoilov, I., To, T.M., et al.: Risk of blindness among patients with diabetes and newly diagnosed diabetic retinopathy. Diabetes care 44(3), 748–756 (2021)
  * [26] Xia, P., Xu, D., Ju, L., Hu, M., Chen, J., Ge, Z.: Lmpt: Prompt tuning with class-specific embedding loss for long-tailed multi-label visual recognition. arXiv preprint arXiv:2305.04536 (2023)
  * [27] Xia, P., Yu, X., Hu, M., Ju, L., Wang, Z., Duan, P., Ge, Z.: Hgclip: Exploring vision-language models with graph representations for hierarchical understanding. arXiv preprint arXiv:2311.14064 (2023)
  * [28] Yang, F.E., Cheng, Y.C., Shiau, Z.Y., Wang, Y.C.F.: Adversarial teacher-student representation learning for domain generalization. Advances in Neural Information Processing Systems 34, 19448–19460 (2021)
  * [29] Yang, X., Yao, H., Zhou, A., Finn, C.: Multi-domain long-tailed learning by augmenting disentangled representations. arXiv preprint arXiv:2210.14358 (2022)
  * [30] Yang, Y., Wang, H., Katabi, D.: On multi-domain long-tailed recognition, imbalanced domain generalization and beyond. In: European Conference on Computer Vision. pp. 57–75. Springer (2022)
  * [31] Yao, H., Wang, Y., Zhang, L., Zou, J.Y., Finn, C.: C-mixup: Improving generalization in regression. Advances in neural information processing systems 35, 3361–3376 (2022)
  * [32] Yao, H., Wang, Y., Li, S., Zhang, L., Liang, W., Zou, J., Finn, C.: Improving out-of-distribution robustness via selective augmentation. In: International Conference on Machine Learning. pp. 25407–25437. PMLR (2022)
  * [33] Yao, H., Yang, X., Pan, X., Liu, S., Koh, P.W., Finn, C.: Improving out-of-domain generalization with domain relations. In: The Twelfth International Conference on Learning Representations (2023)
  * [34] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. In: International Conference on Learning Representations (2018)
  * [35] Zhou, K., Liu, Z., Qiao, Y., Xiang, T., Loy, C.C.: Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
  * [36] Zhou, K., Yang, Y., Hospedales, T., Xiang, T.: Deep domain-adversarial image generation for domain generalisation. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, pp. 13025–13032 (2020)
  * [37] Zhou, K., Yang, Y., Qiao, Y., Xiang, T.: Domain generalization with mixstyle. In: International Conference on Learning Representations (2020)
  * [38] Zhou, Y., Wang, B., Huang, L., Cui, S., Shao, L.: A benchmark for studying diabetic retinopathy: segmentation, grading, and transferability. IEEE Transactions on Medical Imaging 40(3), 818–828 (2020)

Appendix for "Generalizing to Unseen Domains in Diabetic Retinopathy with
Disentangled Representations"

Figure 4: Data distribution for each category in 6 datasets. Figure 5:
Distribution of individual datasets. Table 4: Hyperparameters for experiments.
† denotes we adopt a warm start strategy of running vanilla ERM for the first
few epochs to ensure reliable disentanglement. Interpolation coefficient
$\lambda_{c}\sim\;\text{Beta}(\alpha_{c},\alpha_{c})$ and
$\lambda_{d}\sim\;\text{Beta}(\alpha_{d},\alpha_{d}).$

Hyperparameters | DECO (DG) | DECO (ESDG) | Comparison methods
---|---|---|---
Epochs | 100 | 100 | 100
Learning Rate | $5\times 10^{-4}$ | $5\times 10^{-4}$ | $5\times 10^{-4}$
Weight Decay | $10^{-4}$ | $10^{-4}$ | $10^{-4}$
Batch Size | 32 | 32 | 32
Warm Start Epochs† | 30 | 35 | -
Class Prototype Mixup Parameter $\alpha_{c}$ | 0.5 | 0.55 | -
Domain Prototype Mixup Parameter $\alpha_{d}$ | 0.5 | 0.4 | -
$\gamma_{c}$ | 0.2 | 0.2 | -
$\gamma_{d}$ | 0.2 | - | -


# MaRDIFlow: A CSE workflow framework for abstracting meta-data from FAIR
computational experiments

Pavan L. Veluvali  {veluvali, heiland<EMAIL_ADDRESS>Jan
Heiland  Peter Benner

###### Abstract

Numerical algorithms and computational tools are instrumental in navigating
and addressing complex simulation and data processing tasks. The exponential
growth of metadata and parameter-driven simulations has led to an increasing
demand for automated workflows that can replicate computational experiments
across platforms. In general, a computational workflow is defined as a
sequential description for accomplishing a scientific objective, often
described by tasks and their associated data dependencies. If characterized
through input-output relation, workflow components can be structured to allow
interchangeable utilization of individual tasks and their accompanying
metadata. In the present work, we develop a novel computational framework,
namely, MaRDIFlow, that focuses on the automation of abstracting meta-data
embedded in an ontology of mathematical objects. This framework also
effectively addresses the inherent execution and environmental dependencies by
incorporating them into multi-layered descriptions. Additionally, we
demonstrate a working prototype with example use cases and methodically
integrate them into our workflow tool and data provenance framework.
Furthermore, we show how to best apply the FAIR principles to computational
workflows, such that abstracted components are Findable, Accessible,
Interoperable, and Reusable in nature.

###### keywords:

Computational Workflows, Research Data Management, FAIR Research, Scientific
Computing

This manuscript presents a novel research data management tool, MaRDIFlow ,
that focuses on the automation of abstracting meta-data embedded in an
ontology of mathematical objects. The working prototype of this software tool
is illustrated through FAIR computational experiments.

##  1 Introduction

The interplay of data-intensive computational studies is a substantial part of
scientific endeavors across all disciplines. Computational workflows have been
used as a systematic way of describing the methods needed, the data involved,
as well as computing resources and infrastructures. With ever more complex
simulation models and ever larger primary data volumes, CSE (Computational
Sciences and Engineering) workflow descriptions themselves have become an
enabler for research beyond the execution of simulations, for example, to
extract latent information from various data repositories and compare
methodologies across diverse data and computational frameworks [AGMT17].

The FAIR principles [WDA+16], describe a set of requirements for data
management and stewardship to ensure that the research data are Findable,
Accessible, Interoperable, and Reusable. Each guiding principle is proposed to
define the degree of ‘FAIRness’ via describing the distinct considerations for
contemporary environments, tools, vocabularies and data infrastructures. While
the elements of FAIR Principles are related and separable, they are equally
applied to identify, describe, discover, and reuse meta-data assets of
scholarly outputs. Overall, FAIR principles act as a guide to assist data
stewards in evaluating their implementation choices. More recently, they have
been adopted by funding agencies, such as the German Research Foundation
[For22] for developing assessment metrics of research metadata across various
disciplines [DHM+20].

While there is a knowledge base for CSE workflows from a (software)
engineering point of view [HW09, BCG+19] and while it has been acknowledged
that for documentation, model descriptions and code can complement each other
[FHHS16], an inclusive abstract description of CSE workflows is not yet
anchored. As for combining models, code, and data for the description of CSE
simulations in a virtual lab notebook, Jupyter notebooks have gained
popularity [KRKP+16a]. Also services like Code Ocean [CSFG19] target the
combination of code and model descriptions. Still, little effort has been made
to use abstraction for CSE workflow components in view of documentation tools
that are generally applicable and that scale well with ever more demanding and
sophisticated simulations. Lately, with the advancement of data intensive
research, there has been a rise in the development of automated and reusable
workflows, wherein these workflows aim to seamlessly integrate computer-based
and laboratory computations through artificial intelligence [Nat22].

In this work, we analyze general and particular components and provide an
abstract multi-layered description of CSE workflows: Each component will be
characterized through an input/output description so that model, data, and
code can be used interchangeably and, in the best case, redundantly. For that,
we describe suitable meta data and a low level language for the descriptions
of general CSE workflows [VHB23]. Additionally, we emphasize that the
introduction of redundancy in the representation of models, code, and data
serves as a positive feature for CSE workflows. This redundancy enhances the
robustness of workflows via ensuring compatibility during potential execution
issues. With interchangeable and multi-level components, workflows become more
adaptable and reproducible, contributing to the overall reliability of a
scientific task. Generally, we understand a CSE workflow as a chain of one or
more interconnected models used for simulations. From existing literature, a
CSE workflow is defined as a precise description of a multi-step procedure to
coordinate multiple tasks and their meta-data dependencies [GCBSR+20]. In
workflow systems, each task is represented through the execution of a
computational process, such as, executing a code, calling a command line tool,
accessing a database, submitting a job to a HPC cluster, or executing a data
processing script.

In their general treatment, the following constraints need to be taken into
account

  1. 1.

In particular in a CSE context, each model might be arbitrarily complex and
computationally demanding.

  2. 2.

Often, the particular numerical realization represents a compromise between
accuracy and computational costs.

  3. 3.

Within a workflow, models are likely implemented in different frameworks or
languages.

  4. 4.

In the case of, say, commercial codes that may well be one part of a workflow,
some simulation models might not be fully available but only evaluable through
interfaces.

  5. 5.

Possibly, the actual simulation code is not available at all but only
descriptions and, in the better case, alternative implementations.

Nevertheless, the goal of any CSE workflow framework is to offer a specialized
programming environment that minimizes the efforts required by scientists or
researchers to perform a computational experiment [VHB23]. In general, CSE
workflow description can be categorized into distinct parts or phases, as
listed below, governing its functional operation.

  * •

Composition and abstraction

  * •

Execution

  * •

Meta-data mapping and provenance

Firstly, during composition and abstraction, a CSE workflow is created either
from scratch or from modifying a previously designed workflow, whereby the
user relies on different workflow components and data catalogs. Some of the
well-known methods for editing and composing workflows are either textual or
graphical or mechanism-based semantic models [DGST09]. The workflow then
abstracts software components written by third parties, and handles
heterogeneity via shielding run time incompatibility and complexities.
Secondly, during execution, the workflow components are executed either by a
computational engine or via a subsystem, wherein a static or an adaptive model
is implemented to realize the meta-data. Importantly, repetitive and
reproducible pipelines in order to manage the control and flow of a simulation
are an important aspect of the second phase. Once the workflow is well
defined, all, or portions of the workflow are sent for mapping. Finally, the
data and all associated metadata and provenance information are recorded and
placed in user-defined registries which are then accessed to design a new
workflow description. Through the following stages, CSE workflow description
act as modular building blocks with standardized interfaces, and are generally
linked and run together by a computational framework.

The present article is organized as follows: in the following section, we
discuss the current state of the art in Jupyter notebooks and computational
workflows. Afterwards, we present our research data management tool, namely,
MaRDIFlow, wherein the framework and its usage as a command-line tool is
discussed in detail. Next, we discuss our RDM tool via minimum working
examples. Lastly, we put forward the conclusions and future direction from the
present work.

## Existing Solutions

Reproducibility poses a multifaceted challenge which often demands a
comprehensive examination. In this work, we focus on the specific aspect of
ensuring reproducibility of computational workflows through integrated
software solutions.

Because of the popularity and universality of Jupyter notebooks, we begin with
highlighting the capabilities of the Jupyter environment, which significantly
boost productivity in computational science and mathematics, while also
promoting reproducibility [BTK+21]. In general, Jupyter Notebooks [KRKP+16b]
are accessed through a modern web browser and are typically designed to
support interactive exploration and publishing records of a scientific
computation. Through text and code blocks, it performs a specific computation
and elucidates it in detail. The code within a given Jupyter Notebook is
organized into cells, which in turn allows individuals to modify and execute,
respectively. Also, the output from each cell appears directly below it and is
stored as part of the document. This approach often facilitates a symbiotic
display of code, data, and model descriptions and naturally ensures
replicability of the experiments; [FHHS16]. In this respect, the side by side
appearance of text (for documentation) and code (for the execution) is in line
with the redundant or multi-layered representation of workflows that we want
to achieve but adds to the complexity of such a notebook realization.

What concerns reproducibility, in the strict sense that an experiment could be
reproduced solely by information provided in the notebook, certain design
decisions in the Jupyter notebook like undocumented versions of the imported
libraries let alone the underlying libraries in the backend may stand in the
way. From a more practical perspective, by its linear design of consecutive
cells, Jupyter notebooks are not well suited to handle larger projects. And
although a call of third party code is certainly possible through direct
Julia/python/R interfaces or through the system’s shell, the embedding of
external tools is not a primary and, thus, not a well-defined feature of
Jupyter notebooks. Finally, the reproducibility of a workflow in a Jupyter
notebook hinges on the code only whereas the description is commonly seen as
an add-on. Thus, there is no built-in mechanism that ensures completeness of
the documentation to function as an equivalent or fully valid addition to the
code base. In fact, recent studies have found that a Jupyter notebook per se
is not a strong guarantee for reproducibility; see Refs. [SM24, PMBF21].

Whereas Jupyter notebooks have become a popular tool for implementing,
documenting, and publishing workflows and experiments of moderate complexity,
in computational science and engineering, multiple advanced tools have
addressed the challenge of managing workflows composed of various simulation
codes. Domain specific workflow managers, such as CWL [CAI+22a] and Galaxy
[Com22] are applicable to high-performance computing in general. CWL [CAI+22a]
is widely known for the description of command-line tools and of the workflows
made from these tools. It includes many features, such as, software
containers, resource requirements, workflow-level conditional branching. While
most of the CWL development began in the field of bioinformatics, since 2016,
the CWL standards have been used in other fields, including hydrology, radio
astronomy, geo-spatial analysis, and high-energy physics. On the other hand,
Galaxy [Com22] serves as an accessible browser-based platform for scientific
computing. It facilitates data sharing, analysis, and visualization for
scientists with minimal technical barriers. In addition, integration with
third-party tools like _noWorkflow_ [PMBF17] allows for effective tracking of
provenance, elucidating the relationship between inputs, code, and generated
files.

A container-based approach to modelling workflow components has been followed
in the _functional mockup interface_ (FMI) [BOA+11] development. Here,
arbitrary simulation model are encapsulated in a complete virtual
computational environment (a container) and made accessible through
interfaces. This facilitates an easy exchange of simulation tools (even
without disclosing the source) and can be integrated in workflow designs, as
it is specified in the FMI using _xml_ syntax.

Most of the aforementioned systems have improved the reproducibility of
computational workflows over the last years, and have become the defacto
standard for syntactic interoperability of workflow management systems.
However, each of the systems discussed come with their own limitations.
Namely, CWL often syntax fails to address the user-defined construction and
interaction with other command-line tools once its execution is finished
[CAI+22b]. Likewise, one of the disadvantages of document based workflow
definitions is their static nature, as the exact flow of the workflow must be
known before execution. This specific mechanism inherently imposes constraints
on programming structures that can be utilized, generally via confining
options to either directed acyclic graphs (DAGs) or directed cyclic graphs
(DCGs), especially in cases where loops are accommodated by the markup
language [UHY+21]. Nevertheless, all the aforementioned frameworks store the
data produced, but none with a focus on explicitly recording provenance in
detail, and, in particular, the workflow components are not expressed in a
multi-level framework or abstract objects. On the other hand, customizing the
system for specific requirements may present a significant challenge for end
users, requiring additional effort and resources.

Container-based implementations like in FMI can mitigate the issues with data
provenance or setting up the environment as much as mere replication is
concerned. However, in view reproducibility in different environments or
reusability and adaptation of workflow components, these containers would need
to be equipped with just the same meta-data as any other workflow design tool
would require it.

Based on the aforementioned challenges, it becomes evident that the existing
research landscape for a comprehensive workflow description tailored to
effectively handle mathematical data within a multi-component framework is
lacking. On that account, the present study aims to bridge the existing gap.

##  2 MaRDIFlow

The design principle of MaRDIFlow is to handle the components as abstract
objects described by their input to output behavior and metadata. By means of
the metadata and by matching the I/O interfaces, the objects can be chained
together to form a workflow; see Fig. 1.

Each item then can have different descriptions or realizations that are, in
the best case, equivalent and redundant; see Fig. 2 for an example of such a
vertical dimension in the workflow. We note that the redundancy is meant from
a theoretical perspective. Practically, the different realizations or
representations can be used in different scenarios like compiling a
mathematical description of the workflow or using lookup tables as a shortcut
during a simulation.

Additionally, this multi-level description enhances or even enables
reproducibility in scenarios where, for example, certain software components
of a workflow are not available but will be replaced by data or a description
and vice versa.

  * •

Defined by metadata

  * •

Realized as Fig. 1 and Fig. 2

Level 1: Mathematical Model

Mechanics Acoustics Electromagnetics

Level 2: Simulation Model

FE-Model 2 FE-Model 3 FE-Model 1

Level 3: Surrogate Model

Neural Network ROM Data Base

Figure 1: Generic chain of models that describe the workflow in the simulation
of transformer noise generation and its realization on different levels of
abstraction. Finite Element Model Reduced Order Model Mathematical Model I/O
Data pt… or …pt… or …pt… or … Figure 2: An exemplified vertical multi-level
dimension of a MaRDIflow component: equivalent and preferably redundant
descriptions of a workflow unit.

The working prototype of MaRDIFlow is designed as a command line tool that
enables researchers and users to execute, document and maintain the provenance
for reproduction and replication of computer-based experiments. As shown in
Fig. 3 via a screenshot, MaRDIFlow \--help option provides a help message with
an immediate sense of what MaRDIFlow is. Through the following list we discuss
some of the important arguments required to configure and perform the most
common tasks with our RDM tool.

Figure 3: A screenshot illustrating the help message of our RDM tool,
MaRDIFlow.

  * •

\--workflow-title provides the working title for the workflow, as default we
have, workflow_title = This is a CSE workflow description under MaRDIFlow.

  * •

\--input requires an inputs object file in .json format such that it consists
of all the numerical parameters for the workflow. An example inputs object
file is shown in Fig. 4, where the key value pairs are a valid string
representing the parameter name, and the values found to the right side of the
colon are the absolute values for the given input parameter. As shown in Fig.
4, the JSON object represents the simulation and material parameters which are
then passed on to the required workflow component.

  * •

\--output-directory argument allows the user to specify the desired output
directory. If not provided, then the workflow output is collected in the root
working directory, namely Output.

  * •

MaRDIFlow can also be configured using a .ini config parser file, which
includes both the required default and user-defined sections. It also acts as
a list of parameters that governs how the tool is run and configured.

  * •

\--config example_config_file.ini will execute MaRDIFlow through terminal. An
example configparser file is showcased via a screenshot in Fig. 5.

  * •

\--component argument is necessary to execute the desired workflow component
in the multi-level framework, as previously discussed. In present version,
users can choose between \--math-data or \--math-solver components for
executing a given I/O data or a numerical model, respectively.

  * •

\--display represents the descriptive part of our RDM tool. Herein,
\--display_html=TRUE or \--display_pdf=TRUE will convert the .md into desired
format. An empty string will parse FALSE bool value. The file path for
workflow description in .md file format is passed on to the workflow tool via
\--inputmarkdown flag. For this flag, it is important that the absolute file
path for input as well as for output is provided.

  * •

\--data configures the second component of the workflow description for a
given I/O data. By utilizing \--get-data and \--get-url-data flags, users can
furnish a workflow description for a lookup table or a database, thereby
presenting an alternative approach to a numerical model.

Figure 4: Example inputs object JSON file format with a set of static
parameters required for a specific workflow component. Figure 5: Screenshot of
an example configparser .ini file to initialize and run MaRDIFlow through
command-line.

Overall, the working prototype of MaRDIFlow ideally provides the user with a
specification description, that elucidates the meta-data for a given use case.
In addition, it also consists of a computational part that acts as a mechanism
to call and execute the given meta-data. In order to further understand our
RDM tool, the following minimum working examples implemented in MaRDIFlow are
discussed below.

## Minimum working examples

In the present section, we present some use cases as minimum working examples
to illustrate in detail the working prototype of MaRDIFlow.

### Methanization Reactor

As a working example within the MaRDIFlow framework, a forward solution that
converts $\textrm{CO}_{2}$ to $\textrm{CH}_{4}$ as a result of methanization
[BHBS21] is illustrated. In general, reactor models are crucial for converting
renewable electricity into chemical energy carriers, specifically through
carbon dioxide methanation [BHBS21]. In this study, a reactor model was
examined through a set of nonlinear partial differential equations (PDEs) for
mass and energy balances. The schematic representation of the workflow is
provided in Fig. 6, and we write the governing PDEs as

Figure 6: A screenshot illustrating the workflow description of a
methanization reactor model using the MaRDIFlow tool. As an output,
screenshots of the console and trajectory of the changing reactor load are
shown here from the workflow. In addition, as a final output, the descriptive
part of the tool shows the necessary meta-data to reproduce the use case.

$\epsilon\rho_{gas}\frac{\partial{w}_{\alpha}}{\partial{t}}=-\rho_{gas}\mathbf{v}\,\nabla\cdot{w_{\alpha}}-\nabla\cdot\mathbf{{j_{\alpha}}}+(1-\epsilon)\nu_{\alpha}{\widetilde{r}}$
(1)

$(\rho{c_{p}})_{eff}\frac{\partial{T}}{\partial{t}}=-(\rho{c_{p}})_{gas}\mathbf{v}\cdot\nabla{T}-\nabla\cdot{\dot{{\mathbf{q}}}}+(1-\epsilon)\Delta_{R}{\tilde{H}}\widetilde{r}$
(2)

In the above set of equations, $w_{\alpha}$ is the component of mass fraction
for
$\alpha\in{\textrm{CO}_{2}},\textrm{H}_{2}\textrm{CH}_{4},\text{H}_{2},\textrm{O}$,
$\mathbf{v}$ is the superficial gas velocity, $\rho$ is the gas density,
$\mathbf{j}_{\alpha}$ are dispersive component fluxes, $\nu_{\alpha}$ is
stoichiometric coefficients, $\widetilde{r}$ is the reaction rate of
methanation, $c_{p}$ is the specific heat, $T$ is the temperature,
$\mathbf{\dot{q}}$ and $\Delta_{R}\tilde{H}$ is the heat reaction.
Furthermore, this study revolves around to ensure that the reactor operates at
maximum conversion rates, even when subjected to varying loads. To address
this specific objective, an optimal control problem (OCP) is defined as
follows:

$\zeta(u)=\frac{1}{t_{e}}\int_{0}^{t_{e}}X_{CO_{2}}(t)\,d{t}$ (3)

Here, we express the reactor model as a system of ordinary differential
equations (ODEs), obtained through the finite volume method applied to the PDE
system. Additionally, on states and control we incorporate inequality
constraints. Further initialization and model specific details can be found in
Ref. [BHBS21]. Workflow description for the methanization reactor model is
defined as given below:

  * •

Initialize the system with the given set of governing equations

  * •

Perform a forward simulation with temperature as the input parameter

  * •

Calculate the conversion rates via calculating the change in $CO_{2}$ mass
fraction with time via post-processing

### Spinodal decomposition in a binary A-B alloy

As a second example, let us consider a two-dimensional simulation of the Cahn-
Hilliard equation [CH58] for an A-B alloy. During spinodal decomposition, when
a homogeneous binary alloy is rapidly cooled from a given temperature, the
resulting domain consists of a fine-grained structure of two phases, and over
time, the fine-grained structure coarsens at the expense of smaller particles.
The development of a fine-grained structure from a homogeneous state is
referred to as spinodal decomposition, while the coarsening mechanism is often
defined as Ostwald ripening.

Figure 7: A screenshot illustrating the workflow description of a Cahn-
Hilliard Model using the MaRDIFlow tool. Screenshots of the console and
simulation images as the outputs from the workflow are shown here. In
addition, as a final output, the descriptive part of the tool shows the
necessary meta-data to reproduce the use case.

The schematic representation of the workflow is provided in Fig. 7, and the
set of governing equations required to simulate the phase-separation behavior
between A-B alloy is given below. At first, we define an order parameter $c$
as the concentration of B atom, and the bulk free energy of the system is
defined by

$G=\int_{v}(g_{chem}(c)+g_{grad}(\nabla{c}))dV$ (4)

where $g_{chem}$ and $g_{grad}$ are the chemical free energy and the gradient
energy densities, respectively. In this study, the chemical free energy
density is formulated as:

$g_{chem}=RT[{c}\ln{c}+{1-c}\ln(1-c)]+Lc(1-c)$ (5)

where $L$ is the atomic interaction parameter, and the gradient energy density
is given as:

$g_{grad}=\frac{a_{c}}{2}|\nabla{c}|^{2}$ (6)

where $a_{c}$ is the gradient energy coefficient. Considering the total free
energy of the system decreases with time, the temporal evolution of the order
parameter $c$ is given by:

$\frac{\partial{c}}{\partial{t}}=\nabla\cdot(M_{c}\nabla\frac{\delta{G}}{\delta{c}})=\nabla\cdot(M_{c}\nabla{\mu})$
(7)

where $\mu$ is the diffusion potential of B atom. According to classical
thermodynamics, $\mu$ is generally expressed as:

$\mu=\frac{\delta{G}}{\delta{c}}=RT[\ln{c}-\ln(1-c)]+L(1-2c)-a_{c}\nabla^{2}{c}$
(8)

From 7, $M_{c}$ is the diffusion mobility of B atom, given by:

$M_{c}=\Biggl{[}\frac{D_{A}}{RT}c+\frac{D_{B}}{RT}(1-c)\Biggr{]}c(1-c)=\frac{D_{A}}{RT}\Biggl{[}c+\frac{D_{B}}{D_{A}}(1-c)\Biggr{]}c(1-c)$
(9)

In the above equation, the parameters $D_{A}$ and $D_{B}$ are the the
diffusion coefficients of the respective A and B atoms in the system. Lastly,
herein, the Cahn-Hilliard equation is discretized by simple finite difference
method, 1${}^{\textrm{st}}$-order Euler method is used for time-integration,
and for spatial derivatives the 2${}^{\textrm{nd}}$-order central finite
difference method is implemented. The workflow for the present use-case is
carried out as given below:

  * •

Initialize the bulk free energy and initial local concentration through an
inputs object JSON file.

  * •

The initial configuration of the simulation domain as shown in Fig. 7.

  * •

Pass the required simulation parameters to the workflow component.

  * •

Time evolution of local concentration as well as the phase-separation process
is captured as an output through simulation images.

  * •

Alongside, concentration for various timesteps is collected as an output as
well.

The above workflow can be performed by using MaRDIFlow --config
config_CH_2D.ini in the root directory terminal, and the resulting output
shall be displayed on the screen, similar to Fig. 7. At the end of the
workflow, the phase-separated simulation screenshots along with the
corresponding equilibrium concentration are collected in the user-defined
output directory.

## Summary and Outlook

The practice to perform data and software intensive tasks has been taken hold
by computational workflows. Subsequently, the rapid growth in their uptake and
application on computer-based experiments presents a crucial opportunity to
advance the development of reproducible scientific softwares. As a part of the
MaRDI consortium [MaR21] on research data management in mathematical sciences,
in this work, we presented a novel computational workflow framework, namely,
MaRDIFlow, a prototype that focuses on the automation of abstracting meta-data
embedded in an ontology of mathematical objects while negating the underlying
execution and environment dependencies into multi-layered vertical
descriptions. Additionally, the different components are characterized by
their input and output relation such that they can be used interchangeably and
in most cases redundantly.

The design specification as well as the working prototype of our RDM tool was
presented through different use cases. In the present version, MaRDIFlow acts
a command-line tool such that it enables users to handle the workflow
components as abstract objects described by input to output behavior. At its
core, MaRDIFlow ensures that the output generated is detailed, and a
comprehensive description facilitates the reproduction of computational
experiments. At first we illustrated the conversion rates of CO2 using a
methanization reactor model, and later, we demonstrated the two-dimensional
spinodal decomposition of a virtual A-B alloy using the Cahn-Hilliard model.
Our RDM tool adheres to FAIR principles, such that the abstracted workflow
components are Findable, Accessible, Interoprable and Reusable. Overall, the
ongoing development of MaRDIFlow aims at covering heterogeneous use cases and
act as a scientific tool in the field of mathematical sciences.

Apart from this, we are also working towards developing an Electronic Lab
Notebook (ELN) in order to visualize as well as execute the MaRDIFlow tool.
The ELN will provide researchers with a user-friendly interface to interact
with the tool efficiently and seamlessly. Lastly, although the present
manuscript introduces our RDM tool as a working proof of concept, we plan to
publish a detailed manuscript with technical details and use cases in the near
future.

## Acknowledgments

The authors are supported by _NFDI4Cat_ and _MaRDI_ , funded by the Deutsche
Forschungsgemeinschaft (DFG), project 441926934 “NFDI4Cat – NFDI für
Wissenschaften mit Bezug zur Katalyse” and project 460135501 “MaRDI –
Mathematische Forschungsdateninitiative”.

## Data Availability

Results presented in this work are apart of an ongoing investigation, however
a working prototype with the second use-case is available and documented at
https://doi.org/10.5281/zenodo.10608764

## References

  * [AGMT17] M. Atkinson, S. Gesing, J. Montagnat, and I. Taylor.  Scientific workflows: Past, present and future, 2017.
  * [BCG+19] A. Brinckman, K. Chard, N. Gaffney, M. Hategan, M. B. Jones, K. Kowalik, S. Kulasekaran, B. Ludäscher, B. D. Mecum, J. Nabrzyski, V. Stodden, I. J. Taylor, M. J. Turk, and K. Turner.  Computing environments for reproducibility: Capturing the “whole tale”.  Future Generation Computer Systems, 94:854–867, 2019.
  * [BHBS21] J. Bremer, J. Heiland, P. Benner, and K. Sundmacher.  Non-intrusive time-pod for optimal control of a fixed-bed reactor for co2 methanation.  IFAC-PapersOnLine, 54(3):122–127, 2021.
  * [BOA+11] T. Blochwitz, M. Otter, M. Arnold, C. Bausch, C. Clauß, H. Elmqvist, A. Junghanns, J. Mauss, M. Monteiro, T. Neidhold, et al.  The functional mockup interface for tool independent exchange of simulation models.  In Proceedings of the 8th international Modelica conference, pages 105–114. Linköping University Press, 2011.
  * [BTK+21] M. Beg, J. Taka, T. Kluyver, A. Konovalov, M. Ragan-Kelley, NM. Thiéry, and H. Fangohr.  Using jupyter for reproducible scientific workflows.  Computing in Science & Engineering, 23(2):36–46, 2021.
  * [CAI+22a] M. Crusoe, S. Abeln, A. Iosup, P. Amstutz, J. Chilton, N. Tijanić, H. Ménager, S. Soiland-Reyes, B. Gavrilović, C. Goble, et al.  Methods included: Standardizing computational reuse and portability with the common workflow language.  Communications of the ACM, 65(6):54–63, 2022.
  * [CAI+22b] MR. Crusoe, S. Abeln, A. Iosup, P. Amstutz, J. Chilton, N. Tijanić, H. Ménager, S. Soiland-Reyes, B. Gavrilović, C. Goble, et al.  Methods included: Standardizing computational reuse and portability with the common workflow language.  Communications of the ACM, 65(6):54–63, 2022.
  * [CH58] JW. Cahn and JE. Hilliard.  Free energy of a nonuniform system. i. interfacial free energy.  The Journal of chemical physics, 28(2):258–267, 1958.
  * [Com22] The Galaxy Community.  The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2022 update.  Nucleic Acids Research, 50(W1):W345–W351, 04 2022.
  * [CSFG19] A. Clyburne-Sherin, X. Fei, and SA. Green.  Computational reproducibility via containers in social psychology.  Meta-Psychology, 3, 2019.
  * [DGST09] E. Deelman, D. Gannon, M. Shields, and I. Taylor.  Workflows and e-science: An overview of workflow system features and capabilities.  Future Generation Computer Systems, 25(5):528–540, 2009.
  * [DHM+20] A. Devaraju, R. Huber, M. Mokrane, P. Herterich, L. Cepinskas, J. de Vries, H. L’Hours, J. Davidson, and Angus W.  Fairsfair data object assessment metrics, October 2020.
  * [FHHS16] J. Fehr, H. Heiland, C. Himpe, and J. Saak.  Best practices for replicability, reproducibility and reusability of computer-based experiments exemplified by model reduction software.  AIMS Mathematics, 1(3):261–281, 2016.
  * [For22] Deutsche Forschungsgemeinschaft.  Guidelines for Safeguarding Good Research Practice. Code of Conduct, April 2022.  Available in German and in English.
  * [GCBSR+20] C. Goble, S. Cohen-Boulakia, S. Soiland-Reyes, D. Garijo, Y. Gil, MR. Crusoe, K. Peters, and D. Schober.  Fair computational workflows.  Data Intelligence, 2(1-2):108–121, 2020.
  * [HW09] M. A. Heroux and J. M. Willenbring.  Barely sufficient software engineering: 10 practices to improve your cse software.  In Proceedings of the 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering, pages 15–21, 2009.
  * [KRKP+16a] T. Kluyver, B. Ragan-Kelley, F. Pérez, B. Granger, M. Bussonnier, J. Frederic, K. Kelley, J. Hamrick, J. Grout, S. Corlay, P. Ivanov, D. Avila, S. Abdalla, and C. Willing.  Jupyter notebooks - a publishing format for reproducible computational workflows.  In F. Loizides and B. Schmidt, editors, Positioning and Power in Academic Publishing: Players, Agents and Agendas, pages 87–90, 2016.
  * [KRKP+16b] T. Kluyver, B. Ragan-Kelley, F. Pérez, B. Granger, M. Bussonnier, J. Jonathan Frederic, K. Kelley, J. Hamrick, J. Grout, S. Corlay, P. Ivanov, D. Damián Avila, S. Abdalla, and C. Willing.  Jupyter notebooks – a publishing format for reproducible computational workflows.  In F. Loizides and B. Schmidt, editors, Positioning and Power in Academic Publishing: Players, Agents and Agendas, pages 87 – 90. IOS Press, 2016\.
  * [MaR21] MaRDI.  Mathematic research data initiative, 2021.  URL: https://www.mardi4nfdi.de.
  * [Nat22] National Academies of Sciences, Engineering, and Medicine.  Automated Research Workflows for Accelerated Discovery: Closing the Knowledge Discovery Loop.  The National Academies Press, Washington, DC, 2022.
  * [PMBF17] JF. Pimentel, L. Murta, V. Braganholo, and J. Freire.  noworkflow: a tool for collecting, analyzing, and managing provenance from python scripts.  Proceedings of the VLDB Endowment, 10(12), 2017.
  * [PMBF21] JF. Pimentel, L. Murta, V. Braganholo, and J. Freire.  Understanding and improving the quality and reproducibility of Jupyter notebooks.  Empirical Software Engineering, 26(4):65, 2021.
  * [SM24] S. Samuel and D. Mietchen.  Computational reproducibility of Jupyter notebooks from biomedical publications.  GigaScience, 13:giad113, 01 2024.
  * [UHY+21] M. Uhrin, SP. Huber, J. Yu, N. Marzari, and G. Pizzi.  Workflows in aiida: Engineering a high-throughput, event-based engine for robust and modular computational workflows.  Computational Materials Science, 187:110086, 2021.
  * [VHB23] PL. Veluvali, J. Heiland, and P. Benner.  Mardiflow: A workflow framework for documentation and integration of fair computational experiments.  In Proceedings of the Conference on Research Data Infrastructure, volume 1, 2023.
  * [WDA+16] MD. Wilkinson, M. Dumontier, IJ. Aalbersberg, G. Appleton, M. Axton, A. Baak, N. Blomberg, JW. Boiten, LB. da Silva Santos, PE. Bourne, et al.  The FAIR guiding principles for scientific data management and stewardship.  Scientific data, 3(1):1–9, 2016.