# Enhancing Post-Hoc Explanation Benchmark Reliability for Image
Classification

Tristan Gomez<EMAIL_ADDRESS>Nantes Université, École Centrale
Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France  Harold Mouchère
<EMAIL_ADDRESS>Nantes Université, École Centrale Nantes, CNRS,
LS2N, UMR 6004, F-44000 Nantes, France

(July 2023)

###### Abstract

Deep neural networks, while powerful for image classification, often operate
as ”black boxes,” complicating the understanding of their decision-making
processes. Various explanation methods, particularly those generating saliency
maps, aim to address this challenge. However, the inconsistency issues of
faithfulness metrics hinder reliable benchmarking of explanation methods. This
paper employs an approach inspired by psychometrics, utilizing Krippendorf’s
alpha to quantify the benchmark reliability of post-hoc methods in image
classification. The study proposes model training modifications, including
feeding perturbed samples and employing focal loss, to enhance robustness and
calibration. Empirical evaluations demonstrate significant improvements in
benchmark reliability across metrics, datasets, and post-hoc methods. This
pioneering work establishes a foundation for more reliable evaluation
practices in the realm of post-hoc explanation methods, emphasizing the
importance of model robustness in the assessment process.

##  1 Introduction

I think it goes into a bit of detail too quickly. You should mention the
following points: \- we’re interested in the explanation for image models \-
more particularly we focus on attribution map methods (like Grad-CAM) \- there
are plenty of post-hoc methods and metrics to compare them.

Deep neural networks have emerged as powerful tools for various machine
learning tasks, including image classification. However, their inherent
complexity often renders them as black boxes, making it challenging to
understand their decision-making process. To address this issue, numerous
explanation methods have been developed.

In this work, we are interested in explaining image classification models and,
more specifically, in generating attribution maps to shed light on the
features used by the models. Attribution maps can be generated in two main
ways: (1) by embedding an attribution module within the model itself,
constituting what is known as an attention model, or (2) through generic post-
hoc methods applied to pre-trained models. To evaluate and compare attribution
methods, the explainability community developped various metrics called
faithfulness metrics111In this paper, faithfulness metrics are also called
attribution or fidelity metrics. [1, 7, 14, 4]. These metrics differ in their
functionning but they all rely on a common principle: masking or highlighting
parts of the input image and measuring if the prediction variation reflects
the saliency of these regions, as estimated by the explanation. By quantifying
the faithfulness of explanations, one can construct benchmarks and compare
attributions methods in terms of fidelity to the model’s decision.

However, despite the extensive research on attribution methods and
faithfulness metrics, a critical challenge remains: the lack of reliable
benchmarking due to the inconsistency issues of these metrics. One particular
problematic aspect of these metrics is that the rankings they produce largely
depends on the image on which the explanation is based. Indeed, the metrics
are unreliable to the point where they provide no insight on the studied post-
hoc methods [15]. One potential explanation is that the samples generated to
compute the saliency metrics are out of the training distribution (OOD) and
lead to chaotic model behaviors. This reliability issue is reminiscent of
challenges faced in psychometrics, where tools have been developed to
objectively assess the reliability of tests. One such tool is Krippendorf’s
alpha [10], a statistic designed to determine the degree of agreement among
testers in a given evaluation context.

In particular, this statistic was already used by Tomsett et al. to evaluate
the reliability of post-hoc method benchmarks [15]. Here, we reuse their
protocol and employ it to compare the reliability of the benchmarks obtained
with various training settings.

The goal of this work is to improve the reliability of benchmarking post-hoc
methods for image classification. To do so, we propose several modifications
to the training setting of models used for benchmarks and measure the
reliability of the rankings obtained using Krippendorf’s alpha for each
training setting.

The first proposed modification consist in feeding the model with perturbed
samples during training to improve robustness. More precisely, we study two
types of perturbation: (1) perturbations obtained when computing the metrics
to prevent OOD issues and (2) adversarial perturbations to smooth the model
input space.

Secondly, we replace the commonly used cross-entropy (CE) loss with the focal
loss (FL) proposed by [12] to improve model calibration. The FL encourages the
model to provide consistent confidence levels, such that increasing the masked
proportion of an image, i.e. decreasing the accuracy, corresponds to a drop in
confidence.

Through empirical evaluation, we demonstrate that these modifications and
their combination can significantly improve the reliability of benchmarking
post-hoc methods, as quantified by Krippendorf’s alpha, depending on the
fidelity metric, dataset and post-hoc methods considered. For example, we show
that the proposed modifications can reduce the number of images required to
compute a post-hoc methods benchmark. To the best of our knowledge, this is
the first approach that attempts at improving the reliability of faithfulness
metrics. This work emphasizes the importance of utilizing robust models when
evaluating explanation methods, and we anticipate that future research on
post-hoc methods will benefit from our proposed framework.

In summary, this paper aims to enhance the benchmark reliability of post-hoc
methods for image classification by addressing the challenges associated with
attribution metrics. By leveraging Krippendorf’s alpha and introducing
modifications to the model training process, we establish a more objective and
robust framework for evaluating the performance of explanation methods.

##  2 Related Work

###  2.1 Post-hoc methods

Post-hoc methods can be divided into three groups:

  * •

CAM-based methods. These methods compute a pondered mean of the last features
maps of a convolutional neural network and differ by the method used to
compute the weight of each feature map. The most popular CAM-based method is
Grad-CAM (GC) [gradcam].

  * •

Backpropagation-based methods. Given that CAM-based methods can only be
applied on CNNs, backpropagation-based methods were developped to obtain
explanation from any neural network by backpropagating information from the
output to the input image. Popular examples include Guided Backpropagation
(GP) and Integrated Gradients (IG).

  * •

Perturbation-based methods. Finally, to further expand the class of models
that can be explained, perturbation-based methods were developped and consist
in perturbing the input image to identify the area that contributes the most
to a prediction. Popular examples include Occlusion and RISE.

###  2.2 Faithfulness metrics

As previously mentionned, the explainability community has developped many
metrics to evaluate the faithfulness of explanations [8, 1, 7, 14, 4]. In this
work, we focus on the following metrics. First, we use the Average Drop (AD)
[1] and Average Drop in Deletion (ADD) [7] metrics. AD upscales the
explanation and applies it as a mask on the input image to mask the least
salient regions. The metric is defined as the average drop in the model’s
confidence once the mask is applied. ADD is similarly defined, except that the
mask is reversed to highlight non-salient regions.

We also use the DAUC and IAUC metrics proposed by Petsiuk et al. [14]. The
DAUC metric perturbs the input image by accumulating black patches on the
input image, starting from the least salient regions. Then, the metric is
defined as the area under the curve of the confidence/masked proportion curve.
The intuition is that the model’s confidence should decrease as the masked
proportion increases. Similarly, the IAUC metric starts from a fully blurred
image, and progressively removes the blur from the image.

We also use the variants proposed by Gomez et al. [4]. The DC metric is a
variant of the DAUC metric where the same perturbations are applied, but the
metric is defined as the Pearson’s correlation between the confidence
varations and the saliency scores. Similarly, the IC metric is a correlation-
based variant of IAUC.

Finally, we also propose here non-cumulative variants of the DC and IC
metrics. Concretely, we do not accumulate the black patches on the image for
DC and we do not progressively unblur the whole image for IC, and only reveal
one patch at a time. We name these variants respectively DC non-cumulative
(DC-NC) and IC non-cumulative (IC-NC). Note that one could also define DAUC-NC
and IAUC-NC variants but such variants produce results that are independant of
the chosen explanation. Indeed, DAUC and IAUC evaluate explanations by the
region ordering they produce, and if the perturbations are not accumulated,
the order in which they are applied does not affect the AUC and the metric
value becomes independant of the explanation.

We show in fig. 1 examples of perturbation produced by the various metrics
studied here. Previous work showed that the perturbed samples are actually out
of the training distribution, probably inducing a chaotic behavior of the
model [4]. Hence, we propose to train the model on perturbed samples to
prevent this issue.

(a) Original image and corresponding explanation.

(b) AD

(c) ADD

(d) DAUC/DC

(e) IAUC/IC

(f) DC-NC

(g) IC-NC

Figure 1: Examples of perturbations produced by the various metrics studied
here.

###  2.3 Krippendorf’s alpha

Tomsett et al. [15] proposed the utilization of Krippendorf’s alpha to assess
the reliability of saliency metrics in image classification. Their study
revealed significant inconsistencies in the rankings produced by these
metrics. Building upon their work, we adopt their framework of using
Krippendorf’s alpha and propose modifications to the standard training setting
to enhance benchmark reliability.

Krippendorf’s alpha [10] is a statistic developed by Krippendorf et al. to be
applied in the psychometrics domain. Following the analogy of Tomsett. et al.,
a collection of saliency methods can be seen as a battery of psychometric
tests administered to the model.

Each test is scored by each saliency metrics and each input image can be
viewed as a distinct rater who administers the battery of tests. Krippendorf’s
alpha then quantifies the degree of agreement among the raters, which in this
case are the input images. Concretely, given a set of N test images, we
generate M explanations for each image using different methods, resulting in a
total of M explanations per image. Using a saliency metric, we assess the
faithfulness of these explanations. By computing one explanation ranking for
each image, we obtain a collection of N rankings.

The level of agreement among these rankings is then quantified using
Krippendorf’s alpha. The formula for calculating Krippendorf’s alpha is as
follows:

$\alpha=1-\frac{{D_{o}}}{{D_{e}}},$

where $D_{o}$ is the observed disagreement, and $D_{e}$ is the expected
disagreement. The observed disagreement $D_{o}$ is the average proportion of
disagreements or differences observed among the raters (i.e. images). It is
computed based on the actual ratings or evaluations provided by the raters.
The expected disagreement $D_{e}$ is the hypothetical disagreement that would
occur by chance. It is calculated based on the distribution of explanation
evaluations and takes into account the marginal distributions of the raters.

Perfect agreement is indicated by $\alpha=1$, while $\alpha=0$ indicates that
the agreement similar to the one occuring by chance. Note that $D_{o}$ can
also be superior to $D_{e}$, resulting in negative values of $\alpha$. Thus,
Krippendorf’s alpha ranges from $-\infty$ to 1.

###  2.4 Focal loss

In [12], Mukhoti et al. proposed an adaptive variant of the FL to improve
neural network calibration. The authors show that the FL put less emphasis on
easy samples, preventing the model from being overconfident on these samples.
The FL is defined as follows:

$\mathcal{L}_{focal}=-(1-p)^{\gamma}\log(p),$

where $p$ is the model’s confidence for the ground truth class and $\gamma$ is
a hyperparameter. The FL is a variant of the CE loss, where the term
$(1-p)^{\gamma}$ is multiplied to the loss. As a consequence, if a sample is
easy to classify, the model’s confidence $p$ will be high and the loss will be
low. Mukhoti et al. provides a method to automatically compute the $\gamma$
parameter depending on the confidence on each sample and show that this
adaptive FL variant significantly improves calibration compared to cross
entropy. Given its simplicity, we use this loss in our experiments to
calibrate models.

##  3 Method

###  3.1 Notations.

Let $I\in\mathbb{R}^{H\times W\times 3}$ be the image passed to the model, and
$y_{c}$ be the score of the ground truth class. We also note
$(i,j)\in[0,...,H]\times[0,...,W]$ the spatial position on the input image and
$(i^{\prime},j^{\prime})\in[0,...,H^{\prime}]\times[0,...,W^{\prime}]$ the
spatial position on the feature maps of the last layer.

###  3.2 Training settings

The goal of this work is to improve the reliability of benchmarking post-hoc
methods for image classification. To do so, we use Krippendorf’s alpha not to
compare faithfulness metrics like Tomsett et al. [15] but to compare training
settings. As a consequence, the experiment highlights the importance of model
training when benchmarking post-hoc methods.

The baseline model is trained on unmodified images with CE loss. We compute
perturbed image batches to be fed to the model along the regular batch to
improve its behavior during the computation of faithfulness metrics. We apply
two types of perturbations. First, we generate faithfulness perturbations (FP)
by applying a mask on the image computed using a faithfulness metric-inspired
algorithm. This helps the model to learn how to process perturbations met
during faithfulness metric computation and prevent OOD issues. Secondly,
adversarial perturbations (AP) are obtained with an adversarial attack method.
The purpose is to smooth the input space of the model, which in turn could
help yielding smoother and less noisy score curve when progressively masking
the whole image, as with a multi-step metric for example.

We now define the following three training settings:

  * •

FP: along the regular batch, a perturbed batch is computed by applying a mask
that reproduce faithfulness metrics artifacts.

  * •

AP: it is similar to FP, but the perturbations are computed using an
adversarial attack method.

  * •

FP+AP: combination of the first two perturbations. We compute one FP batch,
one AP batch and a last one which first undergo FP and then AP (FP+AP).

We use the same loss function for each batch and compute the final loss by
simply adding together the individual losses. Finally, for each of the four
aforementionned training settings (including the baseline), we also define a
setting where we replace the CE function with the FL.

The purpose behind incorporating FL is to enhance the model’s calibration,
thereby potentially preventing the score from rising when progressively
masking the entire image. Certainly, when progressively masking out an entire
image, multi-step metrics expects the confidence score to decrease, but this
may not necesseraly happen as usual standard training methods lead to poor
calibration.

We train every possible combination of the proposed modifications. In total,
this amounts to 8 training settings that are summarized in table 1.

Name | Loss for all batches | Regular batch | FP batch | AP batch | FP+AP batch
---|---|---|---|---|---
Baseline | CE | ✓ |  |  |
FP | CE | ✓ | ✓ |  |
AP | CE | ✓ |  | ✓ |
FP+AP | CE | ✓ | ✓ | ✓ | ✓
FL | FL | ✓ |  |  |
FL + FP | FL | ✓ | ✓ |  |
FL + AP | FL | ✓ |  | ✓ |
FL + FP + AP | FL | ✓ | ✓ | ✓ | ✓

Table 1: Batches used for each training setting.

We now describe how to compute an FP, an AP and an FP+AP batches. Each image
in a FP batch is computed using an explanation produced by a post-hoc method.
We note $I$, $I^{FP}$ and $S$ respectively the input image, the FP image and
the saliency map explaining $I$. For each image, we first randomly decide with
equal probability which type of perturbation is going to be applied. We
distinguish four types of perturbations:

  * •

AD-inspired: we apply the upsampled explanation as a mask on the image:

$I^{FP}=\textrm{norm}(\textrm{upsamp}(S))\times I,$ (1)

where $\textrm{norm}(S)$ is the min-max normalization function, defined as
$\textrm{norm}(S)=\frac{S-min(S)}{max(S)-min(S)}$, $\textrm{upsamp}(S)$ is a
function that upsample S to the resolution of I, and $\times$ is the element-
wise product.

  * •

ADD-inspired: we apply the reversed upscaled explanation as a mask on the
image:

$I^{FP}=(1-\textrm{norm}(\textrm{upsamp}(S)))\times I.$ (2)

  * •

DAUC-inspired: we select a random integer $k\in[0,H^{\prime}W^{\prime}-1]$,
and mask the regions of the image corresponding to the top-$k$ pixels of the
explanation with zeros. Let
$(i_{k},j_{k})\in[0,...,H^{\prime}-1]\times[0,...,W^{\prime}-1]$ be the
spatial position of the top-$k$ pixel of the explanation, we define first a
low-resolution mask as follows:

$I^{FP-LR}_{ij}=\begin{cases}0,&\text{if }\exists
k^{\prime}\in[1,k]~{}\text{s.t.}~{}(i,j)=(i_{k^{\prime}},j_{k^{\prime}})\\\
1,&\text{otherwise,}.\end{cases}$ (3)

Then we upsample this mask with nearest-neighbor interpolation to the
resolution of the input image:

$I^{FP}=\textrm{upsamp}_{NN}(I^{FP-LR})$ (4)

  * •

IAUC-inspired: Similarly to DAUC-inspired, we sample an integer $k$ and blur
the whole image except the top-$k$ areas.

Note that to reduce computation, we obtain a rough saliency map $S$ by
averaging the last feature maps of the CNN (post-ReLU). Note that this is not
one of the post-hoc methods we study, but we found it sufficient to obtain
reliability improvements.

The AP batch is computed by the usual adversarial training method: the regular
batch is processed by the model, a perturbation is computed using an
adversarial attack and is applied on the original images. In this work, the
attack is computed using the Projected Gradient Descent (PGD)method [pgd]. The
FP+AP batch is obtained by computing an adversarial attack on the FP batch.
Batch computation is illustrated in fig. 2.

Figure 2: Illustration of the computation of the batches used for the various
training settings.

###  3.3 Evaluation

Once a model from each setting has been trained, we compute explanations using
post-hoc methods. More precisely, we use four method from each class of post-
hoc methods from the litterature which amount to $M=12$ methods in total [5]:

  * •

4 CAM-based methods: CAM, Grad-CAM (GC), Grad-CAM++ (GC++) and Score-CAM (SC).

  * •

4 Backpropagation-based methods: Guided Backpropagation (GP), Integrated
Gradients (IG), SmoothGrad (SG) and varGrad (VG).

  * •

4 Perturbation-based methods: Feature Permutation (FeatPerm), Feature Ablation
(FeatAbl), Occlusion and RISE.

To compute the explanations, we use a set of N randomly chosen images from the
test set. To represent each class equally, we select C images from each class.
For each dataset, we select the smallest C such that the total number of
images at least equal to 100. For example the CUB dataset [18] has 200
classes, which means we must at least select 200 images (one from each class)
to obtain a balanced representation of the classes and have at least 100
images. On the other hand, the CROHNIPI [16] dataset only has 7 classes, which
means we only need to select 15 images per class to obtain at least 100
images.

Then, each explanation is evaluated by each one of the $K=8$ metrics described
in section 2.2. This produces K matrices of size ($N\times M$) containing the
$N$ evaluations of the $M$ post-hoc methods. Following previous work [15], we
convert each row of the matrices into a ranking (i.e. for each image, we sort
methods according to their estimated fidelity), and compute Krippendorf’s
alpha on the rankings with the ordinal difference function to compare ranks
[10]. We obtain $K$ krippendorf’s alpha values for each training setting.
Finally we measure the alpha variation by comparing the $K$ values with the
ones obtained with the baseline model. If the variation is positive, is means
that the modified training setting yield higher inter-raters aggreement, i.e.
more consistent rankings across the $N$ test images, than the baseline
setting. On the other hand, if the variation is negative, it means that the
modified setting generates even more inconsistent rankings than the standard
setting. This workflow is described in fig. 3.

Figure 3: Workflow of the evaluation process.

##  4 Experiments

###  4.1 Datasets

First, we use the popular CUB-200-2011 (CUB) [18], Standford Cars (CARS) [9]
and FGVC Aircraft (AIRCRAFT) [11] datasets. Secondly, we also employ two real-
world fine-grained datasets. The CROHNIPI dataset [16] is a dataset of 3384
images of colonoscopy images belonging to 7 classes (6 pathological classes
and 1 non-pathological class) and the Time-lapse embryo dataset [3]: a dataset
of 59.831 microscopic images of human embryos belonging to 16 differents
classes (the 16 development stages of the embryo). Table 2 shows the number of
images per dataset and the number of images selected for the evaluation.

Dataset | Image number (training/validation+test) | Class number | Selected image | Total image number
---|---|---|---|---
| number per class | for evaluating explanations
CUB-200-2011 | 11788 (5994+5794) | 200 | 1 | 200
Stanford Cars | 16185 (8041+8144) | 196 | 1 | 196
FGVC Aircraft | 10000 (6667+3333) | 100 | 1 | 100
CROHNIPI | 3384 (2781+697) | 7 | 15 | 105
Time-lapse embryo | 59831 (30104+29727) | 16 | 7 | 112

Table 2: Number of selected images per dataset for evaluating explanations.

###  4.2 Implementation details

We use a ResNet-50 architecture for all experiments. Hyper-parameters are
shown in table 3.

Hyper-parameter | Value
---|---
Batch size | 12
Learning rate | 0.001
Optimizer | SGD
Momentum | 0.9
Weight decay | 1e-8
Classification layer dropout | 0.2
Epoch number | 100 (20 for embryo)
Max worse epoch nb | 10 (5 for embryo)
Input size | $224\times 224$
Saliency maps size | $7\times 7$
RISE mask number | 8000
Noise tunnel samples | 30
Table 3: Hyper-parameters used for training the models. The total epoch number
and max worse epoch number are lower for the embryo datasets as it is larger
than the other datasets (60k vs. 10k images).

When possible, we rely on Captum [8] to compute all post-hoc methods except
CAM and RISE, for which we implement our own version, and SC, for which we
utilize Pytorch-GradCAM’s implementation [2]. We also employ an existing
implementation for Krippendorf’s alpha [6] and leverage Scipy for bootstrap
procedures. Models are trained with Pytorch and the code is available on
Github.”

To compute the adversarial attack we use the Projected Gradient Descent method
[pgd] (PGD) with 4 steps, an alpha of $2/255$ and a maximum perturbation size
of $1/255$.

###  4.3 Results

In this section, we evaluate the reliability of post-hoc methods benchmarks
using Krippendorf’s alpha. In particular, we measure the alpha variation
between the baseline setting and each one of the 7 modified training settings.
To evaluate if the alpha variation is significant, we first compute alpha’s
distribution for every setting using bootstrap, i.e. by re-sampling the N rows
of the matrices with replacement. We generate $5000$ bootstrap samples to
estimate the distribution. Then, we first run a Shapiro test on the alpha
distributions of the baseline and evaluated settings. If the test indicates
that either of the distributions is not normal, we use a Mann-Whitney test to
compare the two distributions. Otherwise, we use a Student’s t-test. If we use
a Student’s t-test, we also run a Levene test to check the homogeneity of the
variances. For all tests, we compare the p-value to a 0.05 threshold.

To illustrate the results obtained, we first show Krippendorf’s alpha
variation for the FP+FL training setting when ranking all the $12$ post-hoc
methods in fig. 4. All differences in this figure are statistically
significant.

One can see that the impact of the FP+FL setting compared to the baseline is
overall strongly positive but also depends on the metric and dataset
considered. For example, the increase obtained on the Standford cars dataset
is 28.9 ($7.1\rightarrow 36.0$) when considering the ADD metric whereas a 10.9
decrease ($52.7\rightarrow 41.9$) is observed with the IC-NC metric. On the
other hand, when considering the AD metric on the Stanford cars dataset,
Krippendorf’s alpha decreases ($60.6\rightarrow 49.6$).

Figure 4: Variation of the mean of Krippendorf’s alpha between the baseline
setting and the FL+FP training setting for each metric and each dataset when
ranking all the 12 post-hoc methods. In each square, the value in the center
of the alpha variation. A positive value indicates an increase of
Krippendorf’s alpha. The values at the bottoms separated by ”$\rightarrow$“
show the baseline and modified setting alpha values. Values are multiplied by
100 for clarity.

For a better overall understanding of the impact of the various training
settings used here, we compute the average alpha difference obtained across
datasets and metrics for each training setting and each post-hoc class
considered (Class-Map, Backpropagation (BP), and Perturbations). We constitute
another group of post-hoc methods with one explanation from each class: SC,
RISE and SG, that we note $\\{SC-RISE-SG\\}$. Table 4 shows the mean variation
of the krippendorf’s alpha compared to the baseline performance depending on
the post-hoc group considered. The best training settings seems to be FP+FL,
AP+FL and AP+FL, showing that the three training modification suggested in
this paper seems to be useful. However, the effect is largely dependant on the
post-hoc group considered. For example, FL seems to have a strongly positive
effect when considering all the post-hoc methods, the perturbation-based
methods or the $\\{SC-RISE-SG\\}$ group, but a negligible effect when
considering the class-map-based or backpropagation-based methods. More
interestingly, combining modifications does not necessarily yield better
results than the individual modifications by themselves, depending on the
post-hoc group. This may imply that some modifications have a redundant effect
or that their combination poses training issues. We leave the study of such
issues for future work.

Model | ALL | Class-Map | BP | Perturbations | Recent
---|---|---|---|---|---
FP | 6.129 | 2.512 | 0.325 | 9.039 | 9.284
AP | -1.262 | 4.777 | 3.920 | -0.839 | -3.166
FL | 3.226 | 1.525 | 2.041 | 1.886 | 4.605
FP+FL | 6.664 | 1.644 | 0.438 | 11.131 | 11.872
AP+FL | -0.771 | 7.160 | 4.589 | -1.309 | -3.073
FP+AP | 4.831 | 3.816 | 5.212 | 5.188 | 4.353
FP+AP+FL | 8.346 | 6.157 | 5.295 | 6.387 | 9.845

Table 4: Mean krippendorf’s alpha variation across datasets and metrics
depending on the post-hoc group considered. A positive value indicates an
increase of Krippendorf’s alpha. Values are multiplied by 100 for clarity.

####  4.3.1 Impact on post-hoc benchmarks

In this section we illustrate the impact of the proposed modifications on the
pos-hoc methods benchmarks obtained. More specifically, we focus on the
benchmark of the 4 perturbation-based post-hoc methods using the DAUC metric
on the CUB dataset and compare the baseline and the FL + FP settings. Recall
that we have $N$ test images on which we compute explanations which means that
for one given metric, we obtain $N$ rankings of the post-hoc methods.

(a) Baseline

(b) FL + FP

Figure 5: Distribution of rank position of the perturbation-based methods
according to the DAUC metric.

In fig. 5 we visualize the distribution of rankings for these methods, to
determine which method tends to occupy each of the top-4 rank positions (in
this case, there are four positions as there are four post-hoc methods in the
perturbation-based group). To achieve this, we created histograms that
illustrate the frequency with which each method achieved a particular rank
position across the $N$ test images. For example, we can see in fig. 5(a) that
among the $N$ ranking, approximately $60\%$ attribute the first place to the
RISE method with the baseline training setting and fig. 5(b) shows that this
number increases to more than $80\%$ with the FL+PB setting. Similarly, with
the baseline setting, it is ambiguous which is the top-3 performing method,
whereas with the FL+PB setting, it is clearer that it would be Occlusion. More
generally, the rankings of the FL+PB setting are more similar with each other
than the baseline rankings, which allow to identify the following global top-4
ranking: RISE, FeatAbl, Occlusion and FeatPerm.

Using the training modifications proposed, the explanations rankings are more
similar to each other, which means that one can reduce the number of images
$N$ on which explanations are computed, while maintaining the same top-1
method, for example. Note that reducing $N$ has significant benefits as the
computation cost for post-hoc benchmarks is high. Indeed, simply computing the
explanations has a high cost (particularly with perturbation-based
explanations like RISE which require several thousand inferences per images),
and computing a multi-step metric like DAUC or IAUC which scales quadratically
with the explanation dimension size. Hence, we study in the next section how
much can the test size $N$ be reduced using the new training settings to
further the analysis.

###  4.4 Test size reduction

In this section, we study how much can the number $N$ of images used to
benchmark the post-hoc methods can be reduced while maintaining the same top-1
method. We focus this analysis on the top-1 method for simplicity. We note
$n_{i}$ the number of images on which the post-hoc method $i$ was ranked first
among all the $N$ images. Let $i_{best}$ be the index of the post-hoc method
identified as the best by the benchmark, i.e. such that:

$n_{i_{best}}=\underset{i\in[1,M]}{max}\\{n_{i}\\}.$ (5)

Now, we want to identify the smallest $N^{\prime}\leq N$ such that when
sampling $N^{\prime}$ images among the $N$ without replacement, the method
$i_{best}$ stays the top-1 method. We note $n^{\prime}_{i}$ the number of
images on which method $i$ is the best method among all the $N^{\prime}$
images. Formally, we are looking for the smallest $N^{\prime}$ such that:

$n^{\prime}_{i_{best}}=\underset{i\in[1,M]}{max}\\{n^{\prime}_{i}\\}$ (6)

Note that $n^{\prime}_{i}$ is a random variable that depends on the subset of
images selected. Hence, we search for the smallest $N^{\prime}$ such that the
probability that method $i_{best}$ stays the best method is superior to
$1-\alpha$ where $\alpha$ is a risk that we set to $0.05$:

$P_{N^{\prime}}=P(n^{\prime}_{i_{best}}=\underset{i\in[1,M]}{max}\\{n^{\prime}_{i}\\})\geq
1-\alpha$ (7)

Therefore, we seek to solve the following problem:

$N^{*}=\underset{N^{\prime}\in[1,N]}{min}\left(P_{N^{\prime}}\geq
1-\alpha\right),$ (8)

where we call $N^{*}$ the minimum benchmark size.

This probability can be computed by first listing all possibles outcomes for
the counts $n^{\prime}_{i}$. For example if we benchmark $M=3$ methods on
$N^{\prime}=2$ images, the list of possible outcomes is shown in table 5.

$n^{\prime}_{1}$ | $n^{\prime}_{2}$ | $n^{\prime}_{3}$
---|---|---
0 | 0 | 2
0 | 2 | 0
2 | 0 | 0
0 | 1 | 1
1 | 1 | 0
1 | 0 | 1
Table 5: List of possible outcomes for the counts $n^{\prime}_{i}$ when
benchmarking $M=3$ methods on $N^{\prime}=2$ images.

Secondly, we compute the probability of each outcome using the probability
mass function of the multinomial distribution:

$P(\mathbb{n^{\prime}}_{M})=\begin{cases}\frac{N^{\prime}!}{n^{\prime}_{1}\cdot
n^{\prime}_{2}\cdot...\cdot n^{\prime}_{M}}p_{1}^{n^{\prime}_{1}}\cdot
p_{2}^{n^{\prime}_{2}}\cdot...\cdot
p_{M}^{n^{\prime}_{M}}&\text{if}\sum_{i=0}^{M}n^{\prime}_{i}=N^{\prime}\\\
0,&\text{otherwise},\end{cases}$ (9)

where $\mathbb{n^{\prime}}_{M}$ is a shorthand for the tuple
$(n^{\prime}_{1},n^{\prime}_{2},...,n^{\prime}_{M})$ and where we approximate
the probabilities $p_{i}$ by the corresponding frequency on the full test set
of size $N$: $p_{i}:=f_{i}=n_{i}/N$.

Thirdly, we compute $P_{N^{\prime}}$ by summing all the probabilities of
outcomes where method $i_{max}$ is still ranked as the best method:

$P_{N^{\prime}}=\sum\limits_{\mathbb{n^{\prime}}_{M}\in
Valid(N^{\prime})}P(\mathbb{n^{\prime}}_{M}),$ (10)

where $Valid(N^{\prime})$ is the set of all $(\mathbb{n^{\prime}}_{M})$ tuples
such that $n^{\prime}_{i_{max}}>n^{\prime}_{i},\forall i\neq i_{max}$.

Note that the number of terms in eq. 10 is combinatorialy large, which is why
in practice eq. 10 is approximated by a grid sampling of the tuples. More
precisely, we define a proportion to sample $p_{samp}\in[0,1]$ among the range
of values $[0,N^{\prime}]$ for each $n^{\prime}_{i}$. Then, when listing all
possible tuples, we only evaluate those which satisfy the following condition:

$n^{\prime}_{i}~{}mod~{}1/p_{samp}=0,\forall i\in[1,M]$ (11)

We denote the ensemble of tuples that satisfy this condition
$Sampled(N^{\prime},p)$. In the following experiment, we use $p=0.5$, which
means that we only evaluate tuples with even counts.

Given that we do not evaluate all possible combinations, we update equation
eq. 10 by normalizing the sum as follows:

$P_{N^{\prime}}=\frac{\sum\limits_{\mathbb{n^{\prime}}_{M}\in
Valid(N^{\prime})\cap
Sampled(N^{\prime},p)}P(\mathbb{n^{\prime}}_{M})}{\sum\limits_{\mathbb{n^{\prime}}_{M}\in
Sampled(N^{\prime},p)}P(\mathbb{n^{\prime}}_{M})},$ (12)

Now that we can compute $P_{N^{\prime}}$, we solve eq. 8 using binary search.
Indeed, $P_{N^{\prime}}$ is an increasing function of $N^{\prime}$ where
$P_{1}=p_{i_{best}}$ and $P_{N}=1$. Finally, given that we used different $N$
values for each dataset, we normalize $N^{*}$ by dividing it by the
corresponding $N$. Hence, instead of $N^{*}$, we systematically use
$r=N^{*}/N$ in the following results.

To obtain a global view, we average the $r$ ratio across datasets and metric
for each model and each method group and show the aggregation results in table
6. Note that we do note compute $r$ on all post-hoc methods at the same time
because of the excessive computation cost. The modifications proposed improve
the minimum benchmark size for every method group especially on the Class-Map
methods where benchmark size can only be reduced up to $47.3\%$ with the
baseline setting, whereas with the FP+FL setting, the size can be further
reduced to $28.5\%$ of the original size on average. However, consistently
with section 4.3, the effect of the proposed training settings depends on the
method group considered.

| Recent | Class-Map | BP | Perturbations
---|---|---|---|---
Baseline | 81.1 | 47.3 | 25.8 | 70.0
FP | 80.9 | 32.9 | 28.0 | 82.8
AP | 79.4 | 56.0 | 36.4 | 55.6
FL | 83.8 | 51.8 | 26.7 | 67.1
FP+FL | 75.5 | 28.5 | 22.7 | 75.6
AP+FL | 77.1 | 60.1 | 34.1 | 54.8
FP+AP | 72.4 | 53.4 | 40.2 | 73.4
FP+AP+FL | 77.4 | 52.6 | 37.1 | 64.1

Table 6: Minimum relative benchmark size $r$ for each model and post-hoc
group.

####  4.4.1 Model evaluation

In this section we verify that the proposed modifications to the baseline
training setting are having the indented effect. Tables 8 and 7 respectively
shows the test accuracy and calibration measured with the AdaECE metric [13]
on unmodified images (in the ”Regular“ columns) and on images perturbed by the
faithfulness metrics (the ’FP’ column). Unsurprisingly, the training settings
with FP yield models with superior accuracy on FP images compared to settings
without FP, which confirms the value of training with FP batches. Also, the
best-calibrated models are obtained following training using the FL (except on
FP images on the time-lapse embryo dataset), showing the benefits of FL.
Finally, applying an adverarial attack on images with FP (i.e. FP+AP) helps to
improve the calibration on the FP images on most datasets, especially on the
time-lapse dataset (respectively $28.1$ and $5.6$ for the baseline and FP+AP
settings).

Training setting | CUB-200-2011 | FGVC-Aircraft | Standford cars | CROHNIPI | Time-lapse embryos
---|---|---|---|---|---
| Regular | FP | Regular | FP | Regular | FP | Regular | FP | Regular | FP
Baseline | 2.6 | 8.3 | 6.2 | 15.9 | 2.3 | 9.1 | 10.1 | 17.7 | 18.2 | 28.1
FP | 8.4 | 8.2 | 9.3 | 6.7 | 4.3 | 4.2 | 8.2 | 7.8 | 11.4 | 9.1
AP | 7.5 | 17.3 | 10.5 | 28.8 | 3.5 | 15.6 | 5.3 | 17.8 | 1.9 | 8.4
FL | 4.0 | 9.6 | 1.8 | 14.9 | 1.3 | 7.0 | 3.2 | 5.9 | 10.1 | 18.1
FP+FL | 4.1 | 3.5 | 3.3 | 3.9 | 1.3 | 1.5 | 4.3 | 6.1 | 7.9 | 5.7
AP+FL | 2.6 | 10.1 | 4.4 | 18.8 | 1.0 | 10.1 | 5.2 | 12.5 | 1.5 | 12.6
FP+AP | 9.5 | 8.8 | 10.3 | 10.5 | 4.0 | 3.0 | 4.5 | 12.8 | 9.6 | 5.6
FP+AP+FL | 4.8 | 3.6 | 6.1 | 4.0 | 1.0 | 2.7 | 6.4 | 9.7 | 8.8 | 6.2

Table 7: Calibration error (AdaECE) of the models trained with the 8 studied
training settings. Calibration is measured on unmodified images (in the
”Regular“ columns) and on images perturbed by the faithfulness metrics (the
’FP’ column)

Training setting | CUB-200-2011 | FGVC-Aircraft | Standford cars | CROHNIPI | Time-lapse embryos
---|---|---|---|---|---
| Regular | FP | Regular | FP | Regular | FP | Regular | FP | Regular | FP
Baseline | 76.6 | 44.1 | 79.7 | 48.0 | 87.1 | 56.4 | 83.9 | 73.3 | 63.8 | 40.6
FP | 74.1 | 57.2 | 78.0 | 62.9 | 87.5 | 69.5 | 81.1 | 72.2 | 65.4 | 54.7
AP | 72.7 | 37.0 | 74.3 | 36.5 | 84.0 | 44.5 | 83.5 | 70.4 | 62.3 | 36.4
FL | 79.0 | 43.0 | 79.8 | 43.8 | 87.6 | 52.6 | 83.5 | 66.0 | 63.2 | 42.7
FP+FL | 73.9 | 58.1 | 78.5 | 62.2 | 86.2 | 69.0 | 83.4 | 80.6 | 64.5 | 54.8
AP+FL | 72.0 | 37.3 | 75.4 | 36.4 | 84.1 | 46.0 | 83.2 | 68.1 | 60.5 | 30.5
FP+AP | 71.3 | 47.7 | 76.8 | 57.6 | 86.3 | 66.3 | 83.5 | 73.5 | 64.0 | 51.7
FP+AP+FL | 70.0 | 47.1 | 76.3 | 55.1 | 85.3 | 64.5 | 81.9 | 70.6 | 61.4 | 44.2

Table 8: Accuracy of the models trained with the 8 studied training settings.
Accuracy is measured on unmodified images (in the ”Regular“ columns) and on
images perturbed by the faithfulness metrics (the ’FP’ column)

####  4.4.2 Interpolation

To further understand the impact of the FP on benchmark reliability, we
trained models by interpolating between the regular setting and the FP
setting. More precisely, we trained models with a loss defined as follows:

$\mathcal{L}_{interp}=(2-\beta)\mathcal{L}_{CE}(I,y_{c})+\beta\mathcal{L}_{perturb}(I^{FP},y_{c}),$
with $\beta$ going from 0 to 1. Note that when $\beta=1$, the loss is the same
as the perturbed setting, but when $\beta=0$, the loss is scaled up compared
to the baseline setting. We do this to ensure the global loss value is similar
at every step of the interpolation. Results in fig. 6 show that for the ADD,
DAUC, IAUC, IC and IC-NC there is a clear trend upward when $\beta$ is
increased, i.e., when the training setting become close from the perturbed
loss setting. With the IAUC, IC and IC-NC metrics, $\alpha$’s value is already
close from its final value ($\beta=1$) when $\beta=0.1$. On the other hand,
with the ADD and DAUC metrics, we observe regular increase of $\alpha$’s value
until reaching $\beta=1$. One explanation could be that IAUC, IC and IC-NC all
perturb the image by applying a blur filter, which leaves out the low
frequency information of the image. Hence, blurring is less destructive than
replacing with black patches. Given that the image is less altered and that
information about the original content is still available, it could be easier
for the model to fit the perturbed images.

Figure 6: Krippendorf’s alpha as a function of $\beta$ on the cars dataset.
$\beta=0$ and $\beta=1$ respectively represents the baseline and the perturbed
loss settings. Blue line indicates the mean value and the light blue shade
indicates the $95\%$ confidence interval. Krippendorf’s alpha values are
multiplied by 100 for clarity.

##  5 Discussion

In this study, we proposed a method to improve the reliability of faithfulness
metrics when benchmarking post-hoc explanation methods. By incorporating
perturbed batches and by using an adaptive FL during model training, we
observed significant enhancements in benchmark reliability, as quantified by
Krippendorf’s alpha.

However, we acknowledge that this approach comes with an additional
computation cost. Computation time increases linearly with the number of
batches processed at each step during training. For example, a model trained
in the FP+AP setting has to process the regular batch, the FP batch, and the
AP batch. Despite this additional cost, it is crucial to note that existing
approaches, which apply post-hoc methods on standardly trained models, face
considerable issues in benchmarking explanations from a faithfulness
perspective. Indeed, faithfulness metrics have shown largely inconsistent
rankings when applied to models trained under standard settings, rendering
benchmarking nearly impossible without addressing this issue [15].

Future work could consist in understanding how the modifications proposed
interact with each other and how the final rank reliability can be improved.
For example, one should study whether the AP are redundant with the FP given
that some studies have shown that AP improves OOD generalization [20, 19, 17].

In this work we used Krippendorf’s alpha to measure the post-hoc methods
benchmark consistency across images, following the work of Tomsett et al.
[15]. However, Tomsett et al. also studied the reliability of faithfulness
metrics by measuring the linear correlation between the faithfulness scores
which allow to evaluate the ranking consistency across metrics and post-hoc
methods [15]. Hence, it is essential that future research also include
improving benchmark reliability as measured by the correlation between
fidelity scores.

##  6 Conclusion

This paper presented several simple modifications to model training to enhance
the benchmark reliability of post-hoc explanation methods in image
classification, with a focus on faithfulness metrics. We tackled the issue of
saliency metrics producing out-of-distribution samples by incorporating
adversarial perturbations and perturbations resembling those encountered
during the computation of faithfulness metrics into the training batches.
Additionally, we trained the model using an adaptive Federated Learning (FL)
approach instead of the standard Cross-Entropy (CE) loss. Leveraging
Krippendorf’s alpha, a statistic from psychometrics, we quantifed the
agreement among different saliency metrics, and showed that these training
modifications significantly improves benchmark reliability on various datasets
and faitfhulness metrics. To the best of our knowledge, this study is the
first in the literature to tackle the challenge of improving the reliability
of post-hoc faithfulness benchmarks. In the future, we plan to study benchmark
reliability improvements using other measures than Krippendorf’s alpha like
linear correlation. In conclusion, our research lays the foundation for more
reliable evaluation practices, driving progress in post-hoc explanation
methods and facilitating better interpretability of deep neural networks.

## References

  * [1] A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian.  Grad-cam++ Generalized gradient-based visual explanations for deep convolutional networks.  In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 839–847, 2018.
  * [2] Jacob Gildenblat and contributors.  Pytorch library for cam methods.  https://github.com/jacobgil/pytorch-grad-cam, 2021.
  * [3] Tristan Gomez, Magalie Feyeux, Justine Boulant, Nicolas Normand, Laurent David, Perrine Paul-Gilloteaux, Thomas Fréour, and Harold Mouchère.  A time-lapse embryo dataset for morphokinetic parameter prediction.  Data in Brief, 42:108258, 2022.
  * [4] Tristan Gomez, Thomas Fréour, and Harold Mouchère.  Metrics for saliency map evaluation of deep learning explanation methods.  In Mounîm El Yacoubi, Eric Granger, Pong Chi Yuen, Umapada Pal, and Nicole Vincent, editors, Pattern Recognition and Artificial Intelligence, pages 84–95, Cham, 2022. Springer International Publishing.
  * [5] Tristan Gomez and Harold Mouchère.  Computing and evaluating saliency maps for image classification: a tutorial.  Journal of Electronic Imaging, 32(02), March 2023.
  * [6] Thomas Grill and Santiago Castro.  Python implementation of krippendorff’s alpha – inter-rater reliability.  https://github.com/grrrr/krippendorff-alpha/, 2023.
  * [7] Hyungsik Jung and Youngrock Oh.  Towards better explanations of class activation mapping.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1336–1344, October 2021.
  * [8] Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson.  Captum: A unified and generic model interpretability library for pytorch, 2020.
  * [9] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
  * [10] Klaus Krippendorff.  Content analysis: An introduction to its methodology.  SAGE Publications, Inc., 2455 Teller Road, Thousand Oaks California 91320, 2019.
  * [11] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi.  Fine-grained visual classification of aircraft.  Technical report, 2013.
  * [12] Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip H. S. Torr, and Puneet K. Dokania.  Calibrating deep neural networks using focal loss.  In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc.
  * [13] Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran.  Measuring calibration in deep learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
  * [14] Vitali Petsiuk, Abir Das, and Kate Saenko.  Rise Randomized input sampling for explanation of black-box models, 2018\.
  * [15] Richard Tomsett, Dan Harborne, Supriyo Chakraborty, Prudhvi Gurram, and Alun Preece.  Sanity checks for saliency metrics.  Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):6021–6029, Apr. 2020.
  * [16] Rémi Vallée, Astrid de Maissin, Antoine Coutrot, Harold Mouchère, Arnaud Bourreille, and Nicolas Normand.  CrohnIPI: An endoscopic image database for the evaluation of automatic Crohn’s disease lesions recognition algorithms.  In SPIE Medical Imaging, volume 11317 of Proc. SPIE, Medical Imaging 2020: Biomedical Applications in Molecular, Structural, and Functional Imaging, page 61, Houston, France, February 2020. SPIE.
  * [17] Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John Duchi, Vittorio Murino, and Silvio Savarese.  Generalizing to unseen domains via adversarial data augmentation.  In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 5339–5349, Red Hook, NY, USA, 2018. Curran Associates Inc.
  * [18] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [19] Qixun Wang, Yifei Wang, Hong Zhu, and Yisen Wang.  Improving out-of-distribution generalization by adversarial training with structured priors.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27140–27152. Curran Associates, Inc., 2022.
  * [20] Mingyang Yi, Lu Hou, Jiacheng Sun, Lifeng Shang, Xin Jiang, Qun Liu, and Zhiming Ma.  Improved ood generalization via adversarial training and pretraing.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11987–11997. PMLR, 18–24 Jul 2021.


# KuberneTSN: a Deterministic Overlay Network for Time-Sensitive Containerized
Environments

Andrea Garbugli, Lorenzo Rosa, Armir Bujari, Luca Foschini  University of
Bologna
Department of Computer Science and Engineering
Bologna, Italy
<EMAIL_ADDRESS>
###### Abstract

The emerging paradigm of resource disaggregation enables the deployment of
cloud-like services across a pool of physical and virtualized resources,
interconnected using a network fabric. This design embodies several benefits
in terms of resource efficiency and cost-effectiveness, service elasticity and
adaptability, etc. Application domains benefiting from such a trend include
cyber-physical systems (CPS), tactile internet, 5G networks and beyond, or
mixed reality applications, all generally embodying heterogeneous Quality of
Service (QoS) requirements. In this context, a key enabling factor to fully
support those mixed-criticality scenarios will be the network and the system-
level support for time-sensitive communication. Although a lot of work has
been conducted on devising efficient orchestration and CPU scheduling
strategies, the networking aspects of performance-critical components remain
largely unstudied. Bridging this gap, we propose KuberneTSN, an original
solution built on the Kubernetes platform, providing support for time-
sensitive traffic to unmodified application binaries. We define an
architecture for an accelerated and deterministic overlay network, which
includes kernel-bypassing networking features as well as a novel userspace
packet scheduler compliant with the Time-Sensitive Networking (TSN) standard.
The solution is implemented as tsn-cni, a Kubernetes network plugin that can
coexist alongside popular alternatives. To assess the validity of the
approach, we conduct an experimental analysis on a real distributed testbed,
demonstrating that KuberneTSN enables applications to easily meet
deterministic deadlines, provides the same guarantees of bare-metal
deployments, and outperforms overlay networks built using the Flannel plugin.

###### Index Terms:

time-sensitive networking, container, kubernetes, cloud continuum, network
virtualization, bounded latency

##  I Introduction

The promise of edge computing is that of increasingly low latency, high
bandwidth communication, and improved data security and privacy. Therefore, a
stronger push for edge applications and service deployment is to be expected
[1]. However, in contrast to traditional cloud deployment environments, the
edge has limited resources and may not be able to satisfy the overlapping and
heterogeneous resource demands of all such applications. This fact has
motivated researchers to extend the well-established cloud computing paradigm
into the idea of edge-cloud computing where an increasingly rich and
heterogeneous set of resources between datacenters and the network edge, often
called cloud continuum, can be virtualized to host cloud-like services [2].
The power of this paradigm relies on the combination of the well-known
advantages of the cloud model, in particular flexibility, cost-effectiveness,
and reconfigurability, with the performance advantage of running services as
close to their final user as possible.

The success of this model is clear from its rapid and wide adoption in several
heterogeneous domains, including application domains that embody time-
sensitive requirements. As an example, the reference architecture of 5G and
beyond standards relies on virtualized applications deployed in edge
datacenters, or even co-located with the widely distributed base stations [3].
Control applications in the domains of Cyber-Physical Systems (CPS),
Industrial Internet of Things, Tactile Internet, and in many other fields are
increasingly pursuing the disaggregation trend, with virtualized application
components deployed across the whole continuum of available resources,
embodying heterogeneous Quality of Service (QoS) requirements, even among
their internal components [4, 5]. Although many of those requirements can be
easily met just by placing services physically closer to their final users,
reducing key metrics such as latency or response time, core parts of these
systems still struggle to balance strict performance demand with the overhead
introduced by virtualization.

To mitigate this overhead, lightweight virtualization techniques like
containerization have become the standard technology for platform-independent
prototyping, development, and deployment of edge components. Compared to
hypervisor-based virtual machines, containers are generally characterized by
reduced overhead and higher scalability, representing a potential for
innovation in service patterns, in virtue of setting up a unified service
provisioning platform capable of adhering to applications’ QoS specifications
[6].

Furthermore, containers are seamlessly integrated into resource management and
orchestration platforms, with Kubernetes in its full or reduced versions
(e.g., k3s) as the de-facto standard technology [7]. Resource management and
orchestration are paramount in the edge cloud, as it automatically deploys,
monitors, and migrates containerized application components across the shared
infrastructure, enforcing applications’ QoS specifications.

However, containerization alone is not a panacea. Given the highly distributed
nature of edge cloud applications, specific attention to network and system-
level aspects is paramount to effectively support the most performance-
demanding components. Yet, previous work mostly focused on efficient
orchestration and CPU scheduling of containers [8, 9, 10], leaving those
aspects largely unstudied.

In this paper, we design a cost-efficient solution to enable accelerated and
deterministic communication among containerized applications. To this end, we
define a novel architecture for a container overlay network that combines two
techniques for high-performance communication. First, we adopt a form of
kernel-bypassing networking to remove the overhead of the kernel networking
stack [11]. Second, we propose a novel userspace packet scheduler, compliant
with the Time-Sensitive Networking (TSN) standard, to allow the time-bounded
data distribution and communication among networked components [12]. We
implement our proposal as tsn-cni, a novel Kubernetes network plugin that can
be seamlessly integrated alongside existing options (e.g., Flannel, Calico).
This way, application designers are free to choose the most appropriate
support for traffic flows with different degrees of criticality. Finally, we
evaluate tsn-cni on a real testbed, showing that containerized TSN
applications can achieve determinism and performance comparable to bare metal
applications, and better than using the network fabric set up by the popular
Flannel plugin.

##  II Background

This section provides a brief introduction to container overlay networks,
their rationale, and support in the Kubernetes platform. Next, we provide a
concise background on the TSN standard and kernel-bypassing techniques.

###  II-A Container Overlay Networks

Containers generally have four networking modes available: bridge, host,
macvlan, and overlay. The overlay mode is the most popular, especially in
combination with Kubernetes, as it provides better isolation, ease of use, and
security; hence we limit our description to this scheme. In this mode, as
depicted in Fig. 1, containers are connected on an overlay network,
potentially spanning multiple physical nodes even across different networks.
On each container, a virtual network interface is created, to which
applications can assign an arbitrary IP address. This interface is connected
to the outside through a virtual switch, located in the host operating system
kernel, which has two main roles: it works as a network bridge to allow
communication among co-located containers, and it tunnels network traffic
toward the remote container(s) across the physical network. This way,
containers on the same overlay network have an isolated address namespace and
configuration settings, disjoint from the host network or from other overlays.

When using Kubernetes, by default each container has a single network
interface for all the network traffic, including management and control plane
interactions (e.g., with the Kubernetes master). To distinguish among
different traffic classes, the Multus plugin [13] allows attaching additional
interfaces to containers. Multus is a meta-plugin, as it defines a container
network interface (CNI) that other plugins can implement to configure a Layer
3 network fabric and optionally provide additional advanced features. Several
such plugins are available, such as Flannel, Calico, or Weave. Unfortunately,
none of those supports the definition of an accelerated and deterministic
communication channel among containers. Compared to these alternatives, in
this work we design a novel plugin architecture to offer such guarantees. We
still rely on a virtual switch, but we move the sender-side datapath to
userspace and provide a novel packet scheduler compliant with the TSN
standard. This choice allows users to obtain enhanced network performance with
no modifications to application binaries and atop off-the-shelf hardware and
operating systems, without requiring any patches or specific configurations
from the final user.

Figure 1: Container networking in overlay mode.

###  II-B Time-Sensitive Networking

Designed to support soft real-time industrial traffic, the set of standards
grouped under the name of Time-Sensitive Networking aims to introduce
determinism to IEEE 802.1 networks via a set of features, including but not
limited to time synchronization, programmability, etc. [12]. First, TSN
requires that all the communication participants share a unique time
reference, and the IEEE 802.1AS standalone protocol provides an adequate
mechanism to ensure this synchronization [14]. A second key concept in TSN is
packet scheduling. The IEEE 802.1Qbv standard defines a traffic shaper, called
Time-Aware Shaper (TAS), that can prioritize the frames belonging to classes
of traffic with different time criticality. This prioritization is based on
time-aware communication windows, called _time-aware traffic windows_ , that
repeat cyclically. Each window is divided into _time slots_ that can be
associated to different traffic classes: frames belonging to the same class
are buffered until the next opening of the associated time slot. This way, TSN
guarantees bounded latency and jitter to time-critical traffic, as well as no
interference from best-effort traffic.

From a practical viewpoint, to enable this kind of communication, developers
must configure the kernel-based Traffic Controller (TC), which implements a
TAS shaper, to set up the desired number of traffic classes, their priority,
and time slots duration. Then, applications open a datagram socket with the
SO_TXTIME flag, so that they can associate a desired transmission time to each
outgoing message. Unfortunately, there are two obstacles to the adoption of
this standard from containerized environments. First, we noted that some OS
images do not support the SO_TXTIME. Second, the transmission time is never
forwarded outside the container network namespace to the virtual switch. To
overcome these limitations, KuberneTSN intercepts the container TSN traffic
and forwards it to a novel userspace scheduler, responsible to enforce the TAS
shaping. This component, which replaces the Linux-specific kernel-based
scheduler, is the key architectural element that we leverage to provide time-
sensitive networking features to containerized applications, and it is fully
integrated into the tsn-cni Kubernetes plugin.

###  II-C Kernel-bypassing Networking

In a container overlay network, each outgoing packet must cross the networking
stack twice, one in its isolated network namespace and one in the host
namespace, and must also cross through a virtual switch (Fig. 1). The
combination of all these steps adds significant per-packet communication
overhead [15], unacceptable for time-sensitive edge applications.

In recent years, several kernel-bypassing networking approaches, also known as
network acceleration techniques, have emerged to support performance-critical
applications. Among them, the Data Plane Development Kit (DPDK) [16] is an
increasingly popular library that adopts this approach without requiring
special hardware or OS support. DPDK lets applications access a userspace
version of the network device drivers (Poll Mode Drivers) to directly send or
receive Ethernet packets on the network. Applications and drivers exchange
data through a shared memory area registered with the network card for Direct
Memory Access (DMA), thus communication is zero-copy and avoids kernel/user
context changes. This way, communication is much more efficient, and, in
principle, applications in the edge cloud would immensely benefit from the
related performance improvements. However, DPDK exposes a low-level C
interface, very difficult to use and scarcely integrated within virtualization
engines [17].

In KuberneTSN, we accelerate the outgoing container data path using DPDK
transparently to user applications. Specifically, we design KuberneTSN to
bypass the kernel networking stack in the container namespace, sending data
directly to a userspace virtual switch. Then, we adopt a userspace version of
a widely used and open-source virtual switch, Open vSwitch (OVS) [18], which
in turn uses DPDK to bypass the kernel networking stack in the host namespace.

Overall, KuberneTSN combines three well-known networking approaches, namely
overlay networks, TSN scheduling, and kernel-bypassing networking, and
leverages them to offer the option of a deterministic and accelerated inter-
container communication, well integrated into the state-of-the-art Kubernetes
orchestrator and complementary to existing networking approaches for best-
effort traffic.

##  III Related Work

Previous research on the containerization of critical application components
mainly focused on orchestration strategies and CPU scheduling [10, 8]. These
works investigate the best strategies to place components on suitable
resources and ensure that those resources can schedule the execution of
containerized applications according to their requirements. Yet, they never
take network and system-related aspects into account. We consider these works
complementary to our proposal, as we envision that network and computing
resources for edge applications should be orchestrated together.

Despite the importance of networking for edge applications, researchers paid
less attention to the networking requirements of critical applications. Abeni
et al. [19] evaluate different kernel-bypass approaches for inter-container
communications, outlining the great potential of DPDK as network accelerator
compared to the kernel-based approach. However, their contribution is limited
to a framework for performance evaluation.

Slim [15] proposes a solution to reduce the processing overhead on container
overlay networks. At its crux, the proposal avoids processing packets multiple
times on the same host (see Sec. II); instead, it defines a component that
intercepts calls to the socket API and directly translates network addresses
from the overlay into the host namespace (and vice versa). This way, packets
traverse the kernel networking stack only once. SocksDirect [20] uses the same
interception technique to re-route packets on an accelerated kernel-bypassing
datapath, but this is possible only with the host container networking mode.
Both these works introduce the idea of accelerating container inter-
networking, and both show significant performance advantages for a wide range
of applications built on top of them. However, these solutions are not
integrated with standard production-ready technologies such as Kubernetes.
Furthermore, as they target datacenter environments, their focus is on
accelerated support for reliable connection-oriented transport protocols
(TCP), and they do not provide any support for time-sensitive applications
such as TSN, a key requirement for edge applications. In this work, we adopt
similar techniques (socket interception, kernel-bypassing) to accelerate
network operations, but we also provide guarantees on connection determinism
(through TSN) and implement our solution as a plugin for highly standard
development and deployment technologies.

Finally, the use of TSN in virtual environments is a relatively new trend, as
the standard was originally intended for bare-metal industrial applications.
Leonardi et al. [21] first hypothesized this possibility, identifying three
distinct architectural approaches to enhance hypervisor-based virtualization
with time-triggered communication. In a previous work [22], we showed for the
first time on a real testbed that TSN applications can execute in remote
virtual machines, embodying even better performance than bare-metal thanks to
the adoption of kernel-bypassing techniques. In this paper, we target
containerized applications and take a step further by implementing our
solution as a Kubernetes network plugin, thus allowing an application to
select the most appropriate overlay network meeting their requirements.

##  IV KuberneTSN: an Accelerated and Deterministic Overlay Network

Figure 2: The architecture for an accelerated and deterministic overlay
network, implemented as a Kubernetes CNI plugin.

KuberneTSN defines the architecture for a novel accelerated and deterministic
container overlay network, addressing the time-sensitive requirements of
containerized business or control logic. To achieve this goal, we intervene
and modify the packet processing pipeline for the outgoing container traffic
through the use of two novel architectural components: a user library named
LibKTSN, and a daemon named KTSNd. Fig. 2 shows those components and the role
they play in the definition of a new data path for time-sensitive traffic.

LibKTSN exposes the standard POSIX socket interface to the application
binaries. This way, any time the application issues a send operation on a
datagram socket, the library intercepts it and forwards the packets to a
memory area shared with the KTSNd daemon. We are interested in servicing time-
sensitive traffic, so we only capture outgoing transmissions that have an
explicit transmission time, i.e., TSN traffic, with the SO_TXTIME socket
option. Otherwise, packets are forwarded onto the regular data path. This
approach enables TSN networking regardless of the container images, unlike the
currently available alternatives (see Sec. II). LibKTSN is the only component
of our solution that should be present in the application container. We
provide it as a shared library and use the flag LD_PRELOAD to transparently
intercept traffic: hence, no changes are required to the application code.

The KTSNd daemon represents the key component of our proposal, as it works
both as a packet scheduler and a network accelerator. Once it detects a new
packet from an application, KTSNd schedules its actual transmission based on
the application-provided transmission time. Although we design the daemon to
be agnostic to the specific scheduling strategy, by default it works as a
Time-Aware Shaper (TAS) compliant with the IEEE 802.1Qbv standard (see Sec.
II). Currently, this packet scheduling option is not available for
containerized applications, as popular virtual switches (e.g., Linux bridge,
Open vSwitch, etc.) do not support it. Therefore, our solution is the first to
provide deterministic packet scheduling for unmodified application binaries
running in containers.

When time comes to transmit a scheduled packet, the scheduler must send it on
the network on behalf of the original application, preserving the source MAC,
IP addresses, and UDP ports, and minimizing the packet processing delays to
meet the user-required transmission time as precisely as possible. To satisfy
these requirements, we adopt a kernel-bypassing approach and move the entire
transmission pipeline in userspace. This way, we avoid the expensive double-
crossing of the kernel networking stack and the unnecessary user/kernel thread
context switches (see Sec. II) and instead provide our own simple and
efficient implementation of the UDP/IP stack directly within KTSNd, using the
DPDK library to forward packets on the virtual L2 link. This choice allows us
to preserve the original packet metadata, as we can manipulate protocol
headers directly, and significantly reduce the processing overhead. As shown
in Fig. 2, packets are then handled by a userspace virtual switch that, in
turn, should provide its own UDP/IP userspace stack to forward them on the
physical network. In our implementation, we adopt a widely-used, state-of-the-
art userspace virtual switch, Open vSwitch [18], which also uses DPDK for
kernel-bypassing.

The simple yet powerful design makes KuberneTSN easy to integrate into
standard platforms such as the Kubernetes orchestrator in its various
distributions, making it ready to use for critical networked applications
embodying stringent requirements. To this aim, we build a Kubernetes network
plugin, tsn-cni, that implements our architecture. Specifically, tsn-cni
implements the Multus CNI interface [13] and thus a Layer 3 network fabric
that includes our accelerated and deterministic data path. The plugin requires
applications to include LibKTSN in their execution environment, and it
encapsulates the KTSNd daemon in a separate container. This approach is
strategic to support time-sensitive edge applications: because multiple
network plugins can be used at the same time, developers can choose standard
ones (e.g., Flannel, Calico) for best-effort traffic, and tsn-cni for time-
sensitive networking, as represented in Fig. 2. Therefore, KuberneTSN and its
tsn-cni implementation enhance the capabilities of the edge-cloud not only by
supporting deterministic networking but also by integrating this option in a
familiar ecosystem for application designers. By tagging application
components as time-sensitive, they can instruct Kubernetes to automatically
deploy KTSNd alongside the application containers, thus transparently
obtaining support for performance-sensitive workloads.

642561024Payload
(bytes)$\displaystyle{0}$$\displaystyle{10}$$\displaystyle{20}$$\displaystyle{30}$$\displaystyle{40}$$\displaystyle{50}$Latency
($\displaystyle\mu$s)Bare-metalTSN-CNIFlannel

(a) Latency.

642561024Payload
(bytes)$\displaystyle{-4}$$\displaystyle{-2}$$\displaystyle{0}$$\displaystyle{2}$$\displaystyle{4}$Jitter
($\displaystyle\mu$s)Bare-metalTSN-CNIFlannel

(b) Jitter.

Figure 3: Performance comparison among three deployment options for the
latency test application: bare metal, containerized with tsn-cni,
containerized with Flannel. The experiment is repeated for increasing payload
sizes: $64\text{\,}\mathrm{B}$, $256\text{\,}\mathrm{B}$,
$1024\text{\,}\mathrm{B}$.

##  V Experimental evaluation

In this section, we evaluate the performance of the tsn-cni plugin, which
implements the KuberneTSN architecture. The purpose of the experimental
assessment is twofold: on the one hand, we want to show that the accelerated
datapath we propose is indeed faster than the current state-of-the-art
networking options; on the other hand, we demonstrate that our solution can in
fact provide deterministic guarantees. In particular, we compare tsn-cni
against two alternatives. The first is a bare metal setting that reproduces
the way typical TSN applications are deployed, in order to assess the overhead
introduced by the virtualization layer. The second is Flannel, a popular CNI
plugin for Kubernetes. In its recommended configuration, Flannel uses a Linux
bridge in combination with VXLAN encapsulation to implement the virtual
switch, thus building an overlay network that corresponds to the regular
datapath of Fig. 2. By comparing tsn-cni and Flannel, we assess whether
KuberneTSN meets its design goal of providing additional performance benefits
and deterministic properties to inter-container networking.

For the purpose of this evaluation, we build a simple TSN application
consisting of two processes, a talker and a listener, each running inside a
container on two remote hosts. We then set up a latency test in which the
talker sends UDP packets with a cycle of $1\text{\,}\mathrm{ms}$. The test
measures two representative indicators of time-sensitive communications: end-
to-end latency and jitter. The end-to-end latency of a message is defined as
the time interval between the time of transmission predicted by the talker,
sometimes also called transmission time, and the time of actual reception by
the listener. The jitter measures how much the actual arrival time of each
message differs from the expected arrival time: more precisely, if $t_{i}$ is
the arrival time of the $i$-th message, its jitter is defined as
$Jitter(i)=t_{i}-(t_{i-1}+T)$, where $T$ is the transmission period (in this
work, $T=1ms$). It is noteworthy to point out that the bare-metal and the tsn-
cni test suites are implemented as actual TSN applications, which associate a
desired transmission time to each packet. However, for the test adopting
Flannel as a communication choice, this option is not available, as the TSN
scheduling would not be enforced (see Sec. II). Instead, the only alternative
is to send one message and then sleep, repeating this behavior every $T$.

###  V-A Experimental Settings

The evaluation analysis is conducted on a real testbed which reproduces an
edge deployment scenario. The testbed comprises two Dell Workstations, each
equipped with an Intel I225 NIC, an Intel i9-10980XE 18/36 CPU, and
$64\text{\,}\mathrm{GB}$ RAM. The two hosts are interconnected through a
physical TSN-compliant switch. Each host runs Ubuntu 22.04 with Linux kernel
5.16. When using Open vSwitch [18], we adopt its two variants, the kernel-
bypassing on the sender side, and the kernel-based on the receiver side. As
required by TSN, the clocks of the two hosts are synchronized using two PTP
daemons. Finally, we pin the processes to dedicated cores so to avoid any bias
in the measurements induced by the CPU scheduling policy.

###  V-B End-to-end Latency

Figure 3(a) reports the end-to-end latency and jitter measured for three
typical data sizes ($64\text{\,}\mathrm{B}$, $256\text{\,}\mathrm{B}$,
$1024\text{\,}\mathrm{B}$) for each of the considered deployment scenarios:
bare-metal and containerized applications with tsn-cni or Flannel as network
plugin. A first consideration is that the performance of tsn-cni is always
very good, with median latency values ranging from
$21.5\text{\,}\mathrm{\SIUnitSymbolMicro s}$ in the case of small packets
($64\text{\,}\mathrm{B}$) to $41.7\text{\,}\mathrm{\SIUnitSymbolMicro s}$ for
$1024\text{\,}\mathrm{B}$. These values are almost identical to those
registered for the bare metal deployment, with a small variation in the
$\text{\,}\mathrm{ns}$ scale starting to appear for the 1KB packet size.
Latency variability is negligible in both cases. If we consider Flannel, we
note a slight, but evident latency increase ($12\text{\,}\mathrm{\char
37\relax}$ on average). This is the result of the expensive in-kernel packet
processing, which we avoid thanks to the kernel bypassing technique embodied
in our solution. The same trend observed for latency is confirmed by the
analysis of the jitter metric reported in Fig. 3(b): the median value is zero
in almost all cases and the variability is negligible. Therefore, we can
conclude that KuberneTSN and its tsn-cni implementation succeed in minimizing
the packet processing overhead for containerized applications, achieving the
goal of an accelerated data path.

Overall, our experiments show that both tsn-cni and Flannel show good latency
numbers, although our kernel-bypassing solution shows lower median values. In
principle, one could expect even better performance from tsn-cni, as raw DPDK
is particularly fast [19]. However, we noted that the OVS-DPDK implementation
introduces a non negligible overhead on our userspace datapath, consisting of
at least 23% of the total reported latency. Nevertheless, we decided to keep
it in our system as it is a widely-used tool, supported by an active
community. Even more importantly, while still demonstrating better
performance, it supports a rich set of additional features for virtual
networking, e.g. OpenFlow programmability, compared to the basic Linux bridges
used by Flannel.

###  V-C Determinism

$\displaystyle{0}$$\displaystyle{10}$$\displaystyle{20}$$\displaystyle{30}$$\displaystyle{40}$$\displaystyle{50}$Latency
($\displaystyle\mu$s)$\displaystyle{0.0}$$\displaystyle{0.2}$$\displaystyle{0.4}$$\displaystyle{0.6}$$\displaystyle{0.8}$$\displaystyle{1.0}$..Bare-
metalTSN-CNIFlannel

Figure 4: CDF with packets of 256 bytes.

To assess whether KuberneTSN can effectively provide deterministic guarantees
to time-sensitive flows, we consider again the latency test results discussed
before, but in Fig. 4 we plot the respective Cumulative Distribution Function
(CDF). Ideally, the curve should be as vertical as possible, implying a highly
predictable packet reception time. In this context, the bare metal application
and the containerized application using tsn-cni show overlapping performance,
very close to the ideal behavior. In particular, for tsn-cni the
$90\text{\,}\mathrm{\char 37\relax}$ and the $99\text{\,}\mathrm{\char
37\relax}$ probability correspond to $26.4\text{\,}\mathrm{\SIUnitSymbolMicro
s}$ and $28.1\text{\,}\mathrm{\SIUnitSymbolMicro s}$ respectively. Instead,
for Flannel these thresholds are $29.6\text{\,}\mathrm{\SIUnitSymbolMicro s}$
and $30.7\text{\,}\mathrm{\SIUnitSymbolMicro s}$ respectively, implying a less
precise arrival time interval.

This difference demonstrates the advantage of using KuberneTSN for time-
sensitive traffic. The main reason for this behavior is the way the test
application sends messages: when using Flannel, we cannot explicitly set a
transmission time, as this feature is not supported in current containerized
environments. Hence, we are constrained to fall back to a classic send-and-
sleep loop, mimicking a periodic send operation. The effect of this difference
is minimal in our experiment, as we do not have other competing flows;
however, previous work [23] demonstrates that time-sensitive flows require
dedicated support. tsn-cni serves this purpose by providing essential support
to containerized applications so as to meet heterogeneous flow requirements in
mixed-criticality scenarios.

##  VI Conclusion and future work

We presented KuberneTSN, an architecture for an accelerated and deterministic
container overlay network. KuberneTSN defines a novel userspace TSN packet
scheduler and adopts a kernel-bypassing approach to minimize packet processing
delays. We implemented KuberneTSN as a network plugin for the Kubernetes
orchestrator, called tsn-cni, so that it can be used alongside existing
network fabrics to better support time-sensitive edge applications. The
solution was evaluated on a real testbed, showing that containerized
applications using tsn-cni have the same level of performance and determinism
as bare metal applications, outperforming the widely used Flannel network
plugin.

Future work will include a detailed performance characterization of KuberneTSN
under different traffic conditions, and a demonstration of the use of tsn-cni
in combination with other network plugins. In the longer term, as performance-
demanding AI/ML components are increasingly moved to the network edge, we are
interested in a systematic performance study of the inter-container datapath
to highlight further optimization opportunities.

## Acknowledgements

This work was partially supported by the H2020 TERMINET project (Grant
agreement #: 957406).

## References

  * [1] F.Bonomi, R. Milito, J. Zhu and S. Addepalli, “Fog Computing and Its Role in the Internet of Things,” in _Proc. of the MCC Workshop on Mobile Cloud Computing_. New York, NY, USA: ACM, 2012, p. 13–16.
  * [2] L. Bittencourt, _et al_., “The Internet of Things, Fog and Cloud continuum: Integration and challenges,” _Internet of Things_ , vol. 3-4, pp. 134–155, 2018.
  * [3] P. Trakadas _et al._ , “A Cost-Efficient 5G Non-Public Network Architectural Approach: Key Concepts and Enablers, Building Blocks and Potential Use Cases,” _Sensors_ , vol. 21, no. 16, 2021.
  * [4] Z. Xiang _et al._ , “Reducing Latency in Virtual Machines: Enabling Tactile Internet for Human-Machine Co-Working,” _IEEE Journal on Selected Areas in Communications_ , vol. 37, no. 5, pp. 1098–1116, 2019.
  * [5] R. Ali, Y. B. Zikria, A. K. Bashir, S. Garg and H. S. Kim, “URLLC for 5G and Beyond: Requirements, Enabling Incumbent Technologies and Network Intelligence,” _IEEE Access_ , vol. 9, pp. 67 064–67 095, 2021.
  * [6] A. Randal, “The Ideal Versus the Real: Revisiting the History of Virtual Machines and Containers,” _ACM Comput. Surv._ , vol. 53, no. 1, 2020.
  * [7] Cloud Native Computing Foundation, “Kubernetes.” [Online]. Available: https://kubernetes.io
  * [8] L. Toka, “Ultra-reliable and low-latency computing in the edge with kubernetes,” _Journal of Grid Computing_ , vol. 19, no. 3, pp. 1–23, 2021\.
  * [9] R. Eidenbenz, Y. -A. Pignolet and A. Ryser, “Latency-Aware Industrial Fog Application Orchestration with Kubernetes,” in _Proc. of FMEC_ , 2020, pp. 164–171.
  * [10] J. Harmatos and M. Maliosz, “Architecture Integration of 5G Networks and Time-Sensitive Networking with Edge Computing for Smart Manufacturing,” _Electronics_ , vol. 10, no. 24, 2021.
  * [11] Q. Cai, S. Chaudhary, M. Vuppalapati, J. Hwang and R. Agarwal, “Understanding Host Network Stack Overheads,” in _Proc. of ACM SIGCOMM 2021_ , 2021, p. 65–77.
  * [12] J. Farkas, L. L. Bello and C. Gunther, “Time-Sensitive Networking Standards,” _IEEE Communications Standards Magazine_ , vol. 2, no. 2, pp. 20–21, 2018.
  * [13] Kubernetes Network Plumbing Working Group, “Multus CNI.” [Online]. Available: https://github.com/k8snetworkplumbingwg/multus-cni
  * [14] A. Nasrallah _et al_., “Ultra-Low Latency (ULL) Networks: The IEEE TSN and IETF DetNet Standards and Related 5G ULL Research,” _IEEE Communications Surveys Tutorials_ , vol. 21, no. 1, pp. 88–145, 2019.
  * [15] D. Zhuo _et al._ , “Slim: OS Kernel Support for a Low-Overhead Container Overlay Network,” in _Proc. of USENIX NSDI_ , Boston, MA, 2019, pp. 331–344.
  * [16] Linux Foundation, “Data Plane Development Kit (DPDK),” 2015. [Online]. Available: http://www.dpdk.org
  * [17] L. Rosa and A. Garbugli, “Poster: Insane – a uniform middleware api for differentiated quality using heterogeneous acceleration techniques at the network edge,” in _Proc. of IEEE ICDCS_ , 2022, pp. 1282–1283.
  * [18] B. Pfaff _et al._ , “The Design and Implementation of Open vSwitch,” in _Proc. of USENIX NSDI_ , Oakland, CA, May 2015, pp. 117–130.
  * [19] G. Ara _et al._ , “Comparative Evaluation of Kernel Bypass Mechanisms for High-performance Inter-container Communications,” in _Proc. of the CLOSER_ , 2020, pp. 44–55.
  * [20] B. Li, T. Cui, Z. Wang, W. Bai and L. Zhang, “SocksDirect: Datacenter Sockets can be Fast and Compatible,” in _Proc. of ACM SIGCOMM Conference_ , 2019.
  * [21] L. Leonardi, L. L. Bello and G. Patti, “Towards Time-Sensitive Networking in Heterogeneous Platforms with Virtualization,” in _Proc. of IEEE ETFA_ , vol. 1, 2020, pp. 1155–1158.
  * [22] A. Garbugli, L. Rosa, L. Foschini, A. Corradi and P. Bellavista,, “A Framework for TSN-enabled Virtual Environments for Ultra-Low Latency 5G Scenarios,” in _Proc. of IEEE ICC_ , 2022, pp. 5023–5028.
  * [23] A. Garbugli, A. Sabbioni, A. Corradi and P. Bellavista, “TEMPOS: QoS Management Middleware for Edge Cloud Computing FaaS in the Internet of Things,” _IEEE Access_ , vol. 10, pp. 49 114–49 127, 2022.


# Gravitational frequency shift of light signals in a pulsating dark matter
halo

Vladimir A. Koutvitsky<EMAIL_ADDRESS>Eugene M. Maslov<EMAIL_ADDRESS>Pushkov Institute of Terrestrial Magnetism, Ionosphere and Radio Wave
Propagation RAS, Moscow, Troitsk, Russia

###### Abstract

The gravitational frequency shift of light signals from the center of a
spherically symmetric non-static matter distribution is considered. Explicit
formulas for the ratio of the emitted and received frequencies are obtained in
the case of the oscillating scalar dark matter with logarithmic potential.

dark matter halo, oscillating scalar field, non-static metrics, red shift,
blue shift.

Modern observations show that approximately one fourth of the total mass of
the Universe falls on dark matter, which forms galactic halos and, possibly,
separate self-gravitating lumps Fr . These results are based on the study of
motion of visible matter and propagation of photons in a gravitational field
created by various spatial distributions of dark matter. If the distribution
of dark matter is non–static, then, generally speaking, the corresponding
metric of space–time will be non-static too. In this paper, we investigate the
gravitational frequency shift of a light signal propagating in a spherically
symmetric pulsating lump of scalar dark matter. The corresponding metric will
be

$ds^{2}=B(t,r)\,dt^{2}-A(t,r)\,dr^{2}-r^{2}(d\vartheta^{2}+\sin^{2}\vartheta\,d\varphi^{2}).$
(1)

The general formula for the frequency shift of a photon propagating in a
gravitational field is well known (see, e.g., Schrodinger ). Namely, the ratio
of the source’s proper frequency at the photon emission point to the photon
frequency at the observation point is equal to the ratio of the scalar product
of the photon’s 4–momentum at the emission point and of the source 4-velocity
to the scalar product of the photon’s 4–momentum at the observation point and
of the observer 4–velocity. The 4–momentum of the photon satisfies the
null–geodesic equation, which in the case of a non-static metric reduces to a
rather complicated system of non–linear differential equations with respect to
the affine parameter. Analytical solutions of this system can be obtained only
in certain particular cases, for example, for the metrics describing some
expanding isotropic spacetimes Schrodinger , plane gravitational waves Kauf ;
Far , and small fluctuations of the gravitational potential Ford ; Thom . In
the case of the spherically symmetric metric (1) , the geodesic equation for
the photon 4–momentum was solved numerically in the paper Bosk .

Note that it is possible to obtain a simple formula for the frequency shift in
that case when the source is located at the center of a spherically symmetric
distribution, and the observer is at rest at a point $r=R$. Indeed, setting in
(1) $ds=d\vartheta=d\varphi=0$, we find the equation for the photon trajectory
in the form

$dt/dr=\sqrt{A(t,r)/B(t,r)}.$ (2)

We now consider two light pulses that are successively emitted at the point
$r=0$ through a small time interval $\delta t(0)$. The corresponding close
trajectories, $t(r)$ and $t(r)+\delta t(r)$, satisfy the equation (2) on the
interval $0\leqslant r\leqslant R$. Substituting $t(r)+\delta t(r)$ into (2)
and expanding the right–hand side near the trajectory $t(r)$, we find the
ratio $\delta t(r)/\delta t(0)$. As a result, for the ratio of the proper
frequency of the source $\nu_{0}$ to the observed frequency $\nu_{R}$, we
obtain

$\frac{\nu_{0}}{\nu_{R}}=\sqrt{\frac{B(t(R),R)}{B(t(0),0)}}\exp\int_{0}^{R}\left[\frac{\partial}{\partial
t}\sqrt{\frac{A(t,r)}{B(t,r)}}\right]_{t=t(r)}dr.$ (3)

Note that this formula is also valid in the comoving coordinates: in this
case, in (3), we must put $B=1$. For example, for the Friedmann – Robertson –
Walker metric from (3) it immediately follows that the ratio $\nu_{0}/\nu_{R}$
is equal to the ratio of the scale factors taken at the moments $t(R)$ and
$t(0)$, respectively.

It is easy to verify that the formula (3) can be represented in the following
equivalent form

$\frac{\nu_{0}}{\nu_{R}}=\sqrt{\frac{A(t(R),R)}{A(t(0),0)}}\exp\int_{t(0)}^{t(R)}\left[\frac{\partial}{\partial
r}\sqrt{\frac{B(t,r)}{A(t,r)}}\right]_{r=r(t)}dt.$ (4)

In the post–Newtonian approximation, the formula obtained in Khmel  follows
from this.

Using these results, we calculate the gravitational frequency shift of the
light signal from the center of the self–gravitating pulsating lump of a real
scalar field $\phi(t,r)$ with potential

$U(\phi)=\frac{1}{2}m^{2}\phi^{2}\left(1-\ln\frac{\phi^{2}}{\sigma^{2}}\right),$
(5)

where $\sigma$ is the characteristic magnitude of the scalar field, $m$ is the
mass (in units $\hbar=c=1$). Potentials of this kind arise in the inflationary
cosmology Bar , as well as in some supersymmetric extensions of the standard
model Enq . The corresponding solution of the Einstein–Klein–Gordon system,
describing the self-gravitating pulson, was found in Kout  in the weak field
approximation. It has the form:

$\phi(t,r)/\sigma=[a(\theta)+\varkappa
Q(\theta,\rho)+O(\varkappa^{2})]e^{(3-\rho^{2})/2},$ (6)
$A(t,r)=(1-\rho_{g}/\rho)^{-1},\quad B(t,r)=(1-\rho_{g}/\rho)e^{-s},$ (7)
$\rho_{g}(\tau,\rho)=\varkappa[V_{\max}((\sqrt{\pi}/2)e^{\rho^{2}}\mathrm{erf}\rho-\rho)-a^{2}\rho^{3}]e^{3-\rho^{2}}+O(\varkappa^{2}),$
(8) $s(\tau,\rho)=\varkappa(2V_{\max}+a^{2}\ln
a^{2}+a^{2}\rho^{2})e^{3-\rho^{2}}+O(\varkappa^{2}),$ (9)

where $a(\theta(\tau))$ satisfies the equation of a non–linear oscillator,

$a_{\theta\theta}=-dV/da,\quad V(a)=(a^{2}/2)(1-\ln a^{2}),$ (10)

$\tau=mt$, $\rho=mr$, $\varkappa=4\pi G\sigma^{2}\ll 1$ ($G$ is the
gravitational constant), $V_{\max}=V(a_{\max})$,
$\theta_{\tau}=1+\varkappa\Omega+O(\varkappa^{2})$, $\varkappa\Omega$ is the
pulson frequency correction. The first term in the formula (6) describes the
anharmonic oscillations $-a_{\max}\leqslant a(\theta)\leqslant a_{\max}$ in
the symmetric potential $V(a)$. The function $Q(\theta,\rho)$ is a series of
Hermite polynomials whose coefficients are periodic (in $\theta$) solutions of
the nonhomogeneous Hill equations. In the paper Kout , initial conditions and
the correction $\varkappa\Omega$ were found for which such solutions exist.
The stability of these solutions essentially depends on the amplitude of
oscillations $a_{\max}$. It turned out, that in some intervals of $a_{\max}$
values, solutions with high accuracy retain their periodicity, making hundreds
of oscillations. Similar quasi–stability intervals were also found when
studying the effect of external perturbations on a pulson in the absence of
gravity Kout2 ; Kout3 . Since to calculate the gravitational frequency shift
we need to know only the metric coefficients $A(t,r)$ and $B(t,r)$, we will
not discuss in more detail here the structure of the function
$Q(\theta,\rho)$, whose influence on the metric manifests itself in the next
orders in $\varkappa$, but just assume that $a_{\max}$ belongs to one of the
quasi–stability intervals.

Let the function $a(\theta)$ be an even periodic solution of the equation (10)
with a period

$T(a_{\max})=2\pi/\omega=4\int_{0}^{1}\frac{d\xi}{\sqrt{(1-\ln
a_{\max}^{2})(1-\xi^{2})+\xi^{2}\ln\xi^{2}}}.$ (11)

Then

$\displaystyle a^{2}$ $\displaystyle=$ $\displaystyle
A_{0}/2+\sum_{n=1}^{\infty}A_{n}\cos 2n\omega\theta,$ (12) $\displaystyle
a^{2}\ln a^{2}$ $\displaystyle=$ $\displaystyle
C_{0}/2+\sum_{n=1}^{\infty}C_{n}\cos 2n\omega\theta.$ (13)

Using the equations (10), it is easy to show that the Fourier coefficients
$A_{n}$ and $C_{n}$ are related by

$C_{0}=\frac{A_{0}}{2}-2V_{\max},\quad
C_{n}=\frac{A_{n}}{2}[1-2(n\omega)^{2}],$ (14)

where

$A_{0}=\frac{8a_{\max}^{2}}{T(a_{\max})}\int_{0}^{1}\frac{\xi^{2}d\xi}{\sqrt{(1-\ln
a_{\max}^{2})(1-\xi^{2})+\xi^{2}\ln\xi^{2}}},$ (15)
$A_{n}=\frac{8}{T}\int_{0}^{T/4}a^{2}(\theta)\cos 2n\omega\theta\;d\theta.$
(16)

We now return to the equations (2), (3). Note that due to the smallness of
$\varkappa$, the right-hand side of the equation (2) is close to unity, so
that its time derivative in the equation (3) is of the order $O(\varkappa)$.
Therefore, when calculating the integral in (3), we can take the photon
trajectory in the form $t(r)=r+t(0)$. Thus, substituting (7)-(9) into the
formula (3) and using (12)-(14), we can set
$\theta\approx\tau(\rho)=\rho+\tau(0)$, where
$\tau(0)=\tau(\mathcal{R})-\mathcal{R}$, $\mathcal{R}=mR$. As a result, we
finally get

$\frac{\nu_{0}}{\nu_{R}}=1+z\approx
1+\varkappa\frac{e^{3}}{2}\bigg{\\{}V_{\max}\left(1-\frac{\sqrt{\pi}}{2}\frac{\mathrm{erf}\mathcal{R}}{\mathcal{R}}\right)+\frac{A_{0}}{4}\left(1-e^{-\mathcal{R}^{2}}\right)$
$+\sum_{n=1}^{\infty}\left[C_{n}\left(\cos
2n\omega\tau(0)-e^{-\mathcal{R}^{2}}\cos
2n\omega\tau(\mathcal{R})\right)-2n\omega
P_{n}(\tau(0),\mathcal{R})\right]\bigg{\\}},$ (17)

where

$\displaystyle P_{n}(\tau(0),\mathcal{R})$ $\displaystyle=$
$\displaystyle\big{[}\,C_{n}I_{1}(n\omega,\mathcal{R})-A_{n}I_{3}(n\omega,\mathcal{R})\,\big{]}\cos
2n\omega\tau(0)$ (18)
$\displaystyle+\big{[}\,C_{n}I_{2}(n\omega,\mathcal{R})-A_{n}I_{4}(n\omega,\mathcal{R})\,\big{]}\sin
2n\omega\tau(0),$ $\displaystyle I_{1}(n\omega,\mathcal{R})$ $\displaystyle=$
$\displaystyle\int_{0}^{\mathcal{R}}e^{-\rho^{2}}\sin 2n\omega\rho\;d\rho,$
$\displaystyle I_{2}(n\omega,\mathcal{R})$ $\displaystyle=$
$\displaystyle\int_{0}^{\mathcal{R}}e^{-\rho^{2}}\cos 2n\omega\rho\;d\rho,$
$\displaystyle I_{3}(n\omega,\mathcal{R})$ $\displaystyle=$
$\displaystyle\int_{0}^{\mathcal{R}}\rho^{2}e^{-\rho^{2}}\sin
2n\omega\rho\;d\rho,$ $\displaystyle I_{4}(n\omega,\mathcal{R})$
$\displaystyle=$ $\displaystyle\int_{0}^{\mathcal{R}}\rho^{2}e^{-\rho^{2}}\cos
2n\omega\rho\;d\rho.$ (19)

Integrating by parts, it is easy to show that

$\displaystyle I_{3}(n\omega,\mathcal{R})$ $\displaystyle=$
$\displaystyle\frac{n\omega}{2}+\frac{1}{2}\big{[}1-2(n\omega)^{2}\big{]}I_{1}(n\omega,\mathcal{R})$
$\displaystyle-\frac{1}{2}e^{-\mathcal{R}^{2}}\left(n\omega\cos
2n\omega\mathcal{R}+\mathcal{R}\sin 2n\omega\mathcal{R}\right),$
$\displaystyle I_{4}(n\omega,\mathcal{R})$ $\displaystyle=$
$\displaystyle\frac{1}{2}\big{[}1-2(n\omega)^{2}\big{]}I_{2}(n\omega,\mathcal{R})$
(20) $\displaystyle+\frac{1}{2}e^{-\mathcal{R}^{2}}\left(n\omega\sin
2n\omega\mathcal{R}-\mathcal{R}\cos 2n\omega\mathcal{R}\right).$

Figure 1: Frequency shift versus time at point $\mathcal{R}=0.1$ for different
amplitudes of the halo oscillations: (1) $a_{\max}=0.24$, (2) $a_{\max}=0.44$,
(3) $a_{\max}=0.65$. Figure 2: Frequency shift versus time for $a_{\max}=0.24$
at different distances from the source: (1) $\mathcal{R}=0.5$, (2)
$\mathcal{R}=1$, (3) $\mathcal{R}=1.5$, (4) $\mathcal{R}=3$, (5)
$\mathcal{R}=15$.

At large distances from the source we have

$\displaystyle I_{1}(n\omega,\infty)$ $\displaystyle=$ $\displaystyle
n\omega\,e^{-(n\omega)^{2}}{}_{1}F_{1}\left(\frac{1}{2};\frac{3}{2};(n\omega)^{2}\right),$
$\displaystyle I_{2}(n\omega,\infty)$ $\displaystyle=$
$\displaystyle\frac{\sqrt{\pi}}{2}e^{-(n\omega)^{2}},$ $\displaystyle
I_{3}(n\omega,\infty)$ $\displaystyle=$
$\displaystyle\frac{n\omega}{2}+\frac{n\omega}{2}\big{[}1-2(n\omega)^{2}\big{]}\,e^{-(n\omega)^{2}}{}_{1}F_{1}\left(\frac{1}{2};\frac{3}{2};(n\omega)^{2}\right),$
$\displaystyle I_{4}(n\omega,\infty)$ $\displaystyle=$
$\displaystyle\frac{\sqrt{\pi}}{4}\big{[}1-2(n\omega)^{2}\big{]}\,e^{-(n\omega)^{2}},$
(21)

Then in the function $P_{n}$ (18), in view of (14), (20), the second square
bracket vanishes, so that

$P_{n}(\tau(0),\mathcal{\infty})=-\frac{n\omega}{2}A_{n}\cos 2n\omega\tau(0).$
(22)

As a result, from (17) we find

$\left(\frac{\nu_{0}}{\nu_{R}}\right)_{R\rightarrow\infty}\approx
1+\varkappa\frac{e^{3}}{2}\bigg{\\{}V_{\max}+\frac{A_{0}}{4}+\frac{1}{2}\sum_{n=1}^{\infty}A_{n}\cos
2n\omega\tau(0)\bigg{\\}}.$ (23)

Figures 1 and 2 show the dependence of the frequency shift
$z=(\nu_{0}-\nu_{R})/\nu_{R}$ on the coordinate time at the observation point,
calculated using the formula (17) for different amplitudes of the halo
oscillations $a_{\max}$ and for different distances $R$ from the source. It
turned out that in view of the rapid convergence of the series, when summing
in (17), it suffices to take into account only the first 8 Fourier harmonics.

From the graphs it can be seen that the shift is modulated by the double
frequency of the halo oscillations. The magnitude of the modulations grows
with increasing $a_{\max}$. Note that for some amplitudes and at some
distances, instead of a redshift $(z>0)$, a blue shift $(z<0)$ is observed at
certain time intervals.

## References

  * (1) K. Freese, Proc. 14th Marcel Grossman Meeting; arXiv:1701.01840v1, (2017).
  * (2) E. Schrödinger, Expanding Universes, Cambridge, 1956.
  * (3) W.J. Kaufmann, Nature, 227, (1970), 157.
  * (4) V. Faraoni, Gen. Rel. Grav., 23, (1991), 583.
  * (5) L.H. Ford, Int. J. Theor. Phys., 44, (2005), 1753.
  * (6) R.T. Thompson and L.H. Ford, Phys. Rev. D, 74, (2006), 024012.
  * (7) M. Bošković, F. Duque, M.C. Ferreira, F.S. Miguel, and V. Cardoso, Phys. Rev. D, 98, (2018), 024037.
  * (8) A. Khmelnitsky and V. Rubakov, JCAP, 02, (2014), 019.
  * (9) J. Barrow and P. Parsons, Phys. Rev. D, 52, (1995), 5576.
  * (10) K. Enqvist and J. McDonald, Phys. Lett. B, 425, (1998), 309.
  * (11) V.A. Koutvitsky and E.M. Maslov, Phys. Rev. D, 83, (2011), 124028.
  * (12) V.A. Koutvitsky and E.M. Maslov, Phys. Lett. A, 336, (2005), 31.
  * (13) V.A. Koutvitsky and E.M. Maslov, J. Math. Phys., 47, (2006), 022302.