11institutetext: University of Chinese Academy of Sciences, Beijing, China
11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>22institutetext: Key Laboratory of AI Safety of CAS, Institute of Computing
Technology, Chinese Academy of Sciences (CAS), Beijing, China 22email:
<EMAIL_ADDRESS>
33institutetext: Hangzhou Dianzi University (HDU) & Lishui Institute of HDU,
China

# Distractors-Immune Representation Learning with Cross-modal Contrastive
Regularization
for Change Captioning

Yunbin Tu 11 Liang Li 2*2* Li Su 1*1* Chenggang Yan 33 Qingming Huang 11

###### Abstract

Change captioning aims to succinctly describe the semantic change between a
pair of similar images, while being immune to distractors (illumination and
viewpoint changes). Under these distractors, unchanged objects often appear
pseudo changes about location and scale, and certain objects might overlap
others, resulting in perturbational and discrimination-degraded features
between two images. However, most existing methods directly capture the
difference between them, which risk obtaining error-prone difference features.
In this paper, we propose a distractors-immune representation learning network
that correlates the corresponding channels of two image representations and
decorrelates different ones in a self-supervised manner, thus attaining a pair
of stable image representations under distractors. Then, the model can better
interact them to capture the reliable difference features for caption
generation. To yield words based on the most related difference features, we
further design a cross-modal contrastive regularization, which regularizes the
cross-modal alignment by maximizing the contrastive alignment between the
attended difference features and generated words. Extensive experiments show
that our method outperforms the state-of-the-art methods on four public
datasets. The code is available at https://github.com/tuyunbin/DIRL.

###### Keywords:

Change Captioning Distractors-Immune Representation Learning Cross-modal
Contrastive Regularization

11footnotetext: Corresponding authors

##  1 Introduction

Recently, researchers have witnessed the great success toward vision-language
understanding and generation [45, 22, 31, 44]. As an emerging task, change
captioning [28, 47, 37] is to describe what has semantically changed between
two similar images in natural language (Fig. 1 (a)-(d)). This task has a wide
range of practical applications, such as generating reports for surveillance
area changes [10] and pathological changes between medical images [18], as
well as providing visually-impaired users with the explanations of image
editing effects [30].

Figure 1: The examples of change captioning under different scenarios. The
first and second cases show that object moving and dropping, respectively. The
third one shows the color change under distractors (viewpoint and illumination
changes), where the real change is overwhelmed by pseudo changes. The last one
shows with only distractors. Changed objects are shown in red boxes.

This task also poses a formidable challenge: models should be powerful enough
to not only understand the contents of two images, but also describe the
semantic change between them, while being immune to distractors (viewpoint and
illumination changes). In a dynamic environment, two images of the same scene
are usually obtained in the presence of distractors. In this situation,
unchanged objects between two images often appear obviously pseudo changes
about the scale and location (Fig. 1 (c) (d)), where the features of same
objects within the image pair might be perturbational. Especially, as a
viewpoint drastically changes, certain objects might partially overlap the
others, _e.g.,_ the brown ball is partially occluded by the red block in the
“after” image of Fig. 1 (d). Accordingly, the feature discrimination might be
weakened under this viewpoint. In short, there are perturbational and
discrimination-degraded features between two image representations under
distractors.

Most previous methods directly subtract between such two image representations
[25, 9, 39, 29] or compute their feature similarity [27, 26, 46, 48], which
risk capturing error-prone difference features in between. The latest works
SCORER [36] and SMART [35] overcome the distractors by implementing
contrastive learning between similar/dissimilar image pairs. By maximizing the
alignment of similar ones, both methods make the features of unchanged objects
non-perturbational under distractors. However, both methods disregard the
feature discrimination-degraded problem, leaving the influence of distractors
upon a pair of image representations remained.

In this paper, we propose a Distractors-Immune Representation Learning (DIRL)
network to attain a pair of stable image representations that are non-
perturbational and discriminative under distractors, for robust change
captioning. Concretely, given two raw image representations, DIRL first
computes a channel correlation matrix between them. Next, DIRL implements
cross-channel decorrelation to optimize this matrix to be close to the
identity matrix, _i.e.,_ making the corresponding channels of two image
representations have similar semantics, while enforcing different channels to
be independent. In this self-supervised fashion, DIRL not only alleviates
feature perturbation between two images, but also enhances feature
discrimination for each image. As such, the model can sufficiently interact
the two stable image representations to infer their reliable difference
features, which are then translated into a linguistic sentence via a
transformer decoder.

During sentence generation, as the changed object typically occurs in a local
region with weak feature, traditional attention mechanisms used in mainstream
methods [14, 26, 32] may lead to less satisfactory cross-modal alignment. To
facilitate correct alignment, we further design a Cross-modal Contrastive
Regularization (CCR), which is built upon the cross-attention module of
transformer decoder. When the cross-attention module yields the attended
difference features, CCR further regularizes them by maximizing the
contrastive alignment between them and generated words, which helps the
decoder generate the change caption based on the most related difference
features.

Our key contributions are: (1) We propose the novel DIRL network to learn a
pair of stable image representations under distractors by correlating their
corresponding channels and decorrelating different channels. Based on the two
distractors-immune representations, the model is able to learn the robust
difference features between them for caption generation. (2) We design the CCR
to regularize the cross-modal alignment by maximizing the contrastive
alignment between the features of attended difference and generated words.
This helps the decoder generate words based on the most related difference
features. (3) Our method performs favourably against state-of-the-art methods
on four public datasets with different change scenarios.

##  2 Related Work

Change captioning is a new task in the vision-and-language area [6, 33, 17,
40, 12, 4]. The pioneer work [13] releases the dataset about changes in
surveillance scenarios with potential illumination changes. Afterwards, Tan
_et al._ [30] collect a dataset about image editing scenes. Both works [25,
14] develop two datasets for simulating distractors (viewpoint and
illumination changes), where the unchanged objects’ scale and location would
illustrate obviously pseudo changes.

To learn the difference features under distractors, prior works [25, 9, 21]
directly subtract between two image representations, which are hard to
generalize to unaligned image pairs. SGCC [19] introduces 3D information of
object depths to overcome viewpoint changes. Recent works [27, 14, 48, 32]
first measure the feature similarity to summarize the shared features and then
remove them to obtain difference features. On the other hand, the works [26,
46, 7] correlate the similar features of two images to implicitly deduce the
difference features. However, the above methods do not well address the
influence of distractors upon image representations. The latest works SCORER
[36] and SMART [35] try contrastive learning between similar/dissimilar image
pairs, which model the alignment between similar ones to handle distractors.
There are two differences between both works and our DIRL. First, SCORER and
SMART capture relations between all paired/unpaired images in a batch, whereas
DIRL computes the correlation between the feature channels of each image pair.
Second, SCORER and SMART only make the features between two images non-
perturbational under distractors, while DIRL additionally considers enhancing
feature discrimination for each image to better recognize semantic changes.

During caption generation, since the changed object typically appears in a
local region with weak feature, it is difficult to learn reliable alignment
via current attention modules in prevalent methods. To address this issue,
SMART [35] employs part-of-speech to guide the decoder to dynamically use
visual information. These works [9, 14, 36] propose a cycle consistence module
to enforce correct cross-modal alignment. The other works [46, 7, 3] design a
pre-training and fine-tuning strategy to strengthen the alignment. In this
paper, we regularize cross-modal alignment by proposing CCR, which maximizes
the contrastive alignment between the generated words and attended difference
features. We find that two works in single-image captioning [6, 45] tried the
similar idea. Both works model the coarse-grained cross-modal contrastive
regularization between the input image and generated caption. Instead, our CCR
reuses the attended difference features; establishes the fine-grained
contrastive regularization between them and generated words during decoding.
In this way, the model can use the most relevant difference features to
predict next words.

Figure 2: The framework of our method, where the core blocks are distractors-
immune representation learning network and cross-modal contrastive
regularization. FC and Concat are short for the fully-connect layer and
concatenation operation.

##  3 Methodology

As shown in Fig. 2, the overall framework of our method includes following
parts. First, a pre-trained CNN model extracts the raw representations of an
image pair, which are then transformed into a low-dimensional space. Next, the
two image representations are fed into the proposed DIRL to make them non-
perturbational and discriminative under distractors. Subsequently, a cross-
attention module interacts the two stable representations to mine the shared
features, which are removed to learn the difference features for decoding into
a sentence. During sentence generation, the designed CCR regularizes the
cross-modal alignment, so as to help the decoder generate the sentence based
on the most related difference features.

###  3.1 Image Pair Feature Extraction

Given a pair of images “before” $I_{bef}$ and “after” $I_{aft}$, we first use
a pre-trained CNN to extract their feature representations. They are denoted
as $F_{bef}$ and $F_{aft}$, where ${F}_{o}\in\mathbb{R}^{C\times H\times W}$.
$C$, $H$, $W$ indicate the number of channels, height, and width. Then, we
project them into a low-dimensional space of $\mathbb{R}^{D}$ by a
2D-convolution:

$\tilde{F}_{o}=\text{conv}_{2}({F}_{o})+\text{pos}({F}_{o}),\\\ $ (1)

where $o\in(bef,aft)$. pos is the learnable position encoding function.

###  3.2 Distractors-immune Representation Learning

After obtaining the two position-embedded image representations, we propose a
DIRL network to make them non-perturbational and discriminative under
distractors in a self-supervised manner. Concretely, we first project
$\tilde{F}_{bef}$ and $\tilde{F}_{aft}$ into a common embedding space with
shared parameters:

$\tilde{Y}_{o}=\text{MLP}\left(\tilde{F}_{o}\right),o\in(bef,aft),$ (2)

where MLP is a two-layer multi-layer perceptron with the ReLU activation
function in between. Then, we compute a channel correlation matrix between
$\tilde{Y}_{bef}$ and $\tilde{Y}_{aft}$ along the batch dimension:

$\mathcal{C}_{ij}=\frac{\sum_{b}\tilde{y}_{b,i}^{bef}\tilde{y}_{b,j}^{aft}}{\sqrt{\sum_{b}\left(\tilde{y}_{b,i}^{bef}\right)^{2}}\sqrt{\sum_{b}\left(\tilde{y}_{b,j}^{aft}\right)^{2}}},$
(3)

where $b$ indexes batch samples and $i,j$ index the channel dimension of
features. $\mathcal{C}\in\mathbb{R}^{D\times D}$ is a channel correlation
matrix. Inspired by the recent self-supervised learning methods [49, 43], we
perform cross-channel decorrelation to enforce this matrix to be close to the
identity matrix, which is implemented by using the $\ell_{2}$-norm
minimization:

$\mathcal{L}_{dirl}=\sum_{i}\left(1-\mathcal{C}_{ii}\right)^{2}+\alpha\sum_{i}\sum_{j\neq
i}\mathcal{C}_{ij}^{2}.$ (4)

The first term equates the diagonal of $\mathcal{C}$ to one, i.e., the
corresponding channels of two image representations will be correlated and
thus have similar semantics under distractors. The second term equates the off
diagonal of $\mathcal{C}$ to zero, i.e., the different channels will be
decorrelated. This enhances the discrimination of each image representation.
$\alpha$ is a trade-off parameter to balance the importance between two terms,
which is discussed in the supplementary material.

###  3.3 Difference Representation Learning

The difference features are modeled based on the two distractors-immune image
representations. As shown in Fig. 1, most objects are identical between two
images. Hence, it is natural to extract their shared features to infer the
difference features. For a powerful model, it should capture all potential
changes w.r.t. both images. To this end, we first learn the difference
features for each image and then combine them to construct the difference
features between two images.

Specifically, we first compute the shared features on each image by the multi-
head cross-attention (MHCA) mechanism [41]:

$\begin{gathered}\tilde{F}_{bef}^{s}=\operatorname{MHCA}\left(\tilde{F}_{bef},\tilde{F}_{aft},\tilde{F}_{aft}\right),\\\
\tilde{F}_{aft}^{s}=\operatorname{MHCA}\left(\tilde{F}_{aft},\tilde{F}_{bef},\tilde{F}_{bef}\right).\end{gathered}$
(5)

Subsequently, we subtract each from the corresponding image representation to
compute the difference features on each image, respectively:

$\begin{gathered}\tilde{F}_{bef}^{d}=\tilde{F}_{bef}-\tilde{F}_{bef}^{s},\\\
\tilde{F}_{aft}^{d}=\tilde{F}_{aft}-\tilde{F}_{aft}^{s}.\end{gathered}$ (6)

Both $\tilde{F}_{bef}^{d}$ and $\tilde{F}_{aft}^{d}$ are then concatenated as
the omni-representation of difference features between two images, which is
implemented by a fully-connected layer with the ReLU activation function:

$\tilde{F}_{d}=\operatorname{ReLU}\left(\left[\tilde{F}_{bef}^{d};\tilde{F}_{aft}^{d}\right]W_{c}+b_{c}\right),$
(7)

where [;] is a concatenation operation.

###  3.4 Caption Generation

After learning $\tilde{F}_{d}\in\mathbb{R}^{HW\times D}$, we use a standard
transformer decoder [41] to translate it into a sentence. First, we obtain the
embedding features of $m$ words (ground-truth words during training, predicted
words during inference). Then, we use the multi-head self-attention (MHSA)
layer to model relationships among these word embedding features
$E[W]=\\{E[w_{1}],...,E[w_{m}]\\}$:

${\hat{E}[W]}=\text{ MHSA }(E[W],E[W],E[W]).$ (8)

Subsequently, we use the relation-embedded word features $\hat{E}[W]$ to
attend to the related difference features from $\tilde{F}_{d}$ based on the
MHCA layer:

$\hat{V}=\text{ MHCA }({E[\hat{W}]},\tilde{F}_{d},\tilde{F}_{d}).$ (9)

Next, the $\hat{V}\in\mathbb{R}^{HW\times D}$ is passed to a feed-forward
network (FFN) to obtain the enhanced difference features:

$\hat{V}^{\prime}=\text{ LayerNorm}(\hat{V}+\operatorname{FFN}(\hat{V})).$
(10)

Finally, the probability distributions of target words are calculated via a
single hidden layer:

$W=\operatorname{Softmax}\left(\hat{V}^{\prime}W_{h}+{b}_{h}\right),$ (11)

where $W_{h}\in\mathbb{R}^{D\times\nu}$ and $b_{h}\in\mathbb{R}^{\nu}$ are the
learnable parameters. $\nu$ is the dimension of vocabulary size.

###  3.5 Cross-modal Contrastive Regularization

It is often the case that the changed object appears in a local area with weak
feature, so it is difficult for the cross-attention module to directly align
the captured difference features with word features. To this end, we devise
the CCR to regularize the cross-modal alignment, which reuses the attented
difference features $\hat{V}$ computed in Eq. (9) and maximizes the
contrastive alignment between them and generated words ${\hat{E}[W]}$.
Specifically, we first compute the global representation for ${\hat{E}[W]}$
and $\hat{V}$ via mean-pooling operation, respectively:

$\begin{gathered}\tilde{E}[W]=\frac{1}{m}\hat{E}[W],\\\
\tilde{V}=\frac{1}{HW}\hat{V},\end{gathered}$ (12)

where $m$ and $HW$ are the length of ${\hat{E}[W]}$ and $\hat{V}$. Then, given
a training batch, we sample $B$ pairs of global representations of generated
words and attended difference features. For $k$-th global word representation
$\tilde{E}[W]_{k}$, $k$-th global difference representation $\tilde{V}_{k}$ is
its positive, while the others will be the negatives in this batch. The
similarity between two global representations is computed by dot-product:

$\text{sim}\left(\tilde{E}[W],\tilde{V}\right)=\tilde{E}[W]\cdot\tilde{V}^{\top},$
(13)

Next, we leverage the InfoNCE loss [23] to pull semantically close pairs of
$\tilde{E}[W]_{k}$ and $\tilde{V}_{k}$ together and push away non-related
pairs:

$\begin{gathered}\mathcal{L}_{t2v}=-\frac{1}{B}\sum_{k}^{B}\log\frac{e^{\left(\text{sim}\left(\tilde{E}[W]_{k},\tilde{V}_{k}\right)/\tau\right)}}{\sum_{r}^{B}e^{\left(\text{sim}\left(\tilde{E}[W]_{k},\tilde{V}_{r}\right)/\tau\right)}},\\\
\mathcal{L}_{v2t}=-\frac{1}{B}\sum_{k}^{B}\log\frac{e^{\left(\text{sim}\left(\tilde{V}_{k},\tilde{E}[W]_{k}\right)/\tau\right)}}{\sum_{r}^{B}e^{\left(\text{sim}\left(\tilde{V}_{k},\tilde{E}[W]_{r}\right)/\tau\right)}},\\\
\mathcal{L}_{ccr}=\frac{1}{2}(\mathcal{L}_{t2v}+\mathcal{L}_{v2t}),\end{gathered}$
(14)

where $\tau$ is the temperature hyper-parameter. This loss formulates a self-
supervisory signal to regularize the cross-modal alignment during caption
generation, so as to improve the quality of generated captions.

###  3.6 Joint Training

We train the overall architecture end-to-end by maximizing the likelihood of
the observed word sequence. Given the ground-truth words
$\left(w_{1}^{*},\ldots,w_{m}^{*}\right)$, we minimize the negative log-
likelihood loss:

$\mathcal{L}_{cap}(\theta)=-\sum_{t=1}^{m}\log p_{\theta}\left(w_{t}^{*}\mid
w_{<t}^{*}\right),$ (15)

where $p_{\theta}\left(w_{t}^{*}\mid w_{<t}^{*}\right)$ is computed by Eq.
(11), and $\theta$ are all the learnable parameters. In addition, our method
is self-supervised by the losses of DIRL and CCR. Thus, the total loss is
defined as:

$\mathcal{L}=\mathcal{L}_{cap}+\lambda_{d}\mathcal{L}_{dirl}+\lambda_{c}\mathcal{L}_{ccr},$
(16)

where $\lambda_{d}$ and $\lambda_{c}$ are the trade-off parameters that are
discussed in Sec. 4.6.

##  4 Experiments

###  4.1 Datasets and Evaluation Metrics

Spot-the-Diff [13] has 13,192 image pairs from surveillance cameras, where
each image pair includes an underlying illumination change. According to the
official split, we split it into training, validation, and testing with a
ratio of 8:1:1.

CLEVR-Change [25] has 79,606 image pairs and 493,735 captions. It contains
five change types, _i.e._ , “Color”, “Texture”, “Add”, “Drop”, and “Move”, as
well as moderate distractors. We use the official split of 67,660 for
training, 3,976 for validation, and 7,970 for testing, respectively.

CLEVR-DC [14] has 48,000 image pairs with the same change types as CLEVR-
Change, but this dataset includes drastic distractors. We use the official
split with 85% for training, 5% for validation, and 10% for testing.

Image Editing Request [30] is comprised of 3,939 image pairs with 5,695
editing instructions. We use the official split with 3,061 image pairs for
training, 383 for validation, and 495 for testing, respectively.

Evaluation Metrics: We follow the state-of-the-art methods to use the
following five metrics for evaluating the quality of generated captions:
BLEU-4 [24], METEOR [2], ROUGE-L [20], CIDEr [42] and SPICE [1]. We compute
all the results by the Microsoft COCO evaluation server [5].

###  4.2 Implementation Details

For fair-comparison, we follow current SOTA methods to utilize a pre-trained
ResNet-101 [8] to extract the features of a pair of images, with the dimension
of 1024 $\times$ 14 $\times$ 14\. We project them into a lower dimension of
512. We set the hidden size of the model and word embedding size as 512 and
300. Temperature $\tau$ in Eq. (14) is set to 0.5. We train the model to
converge with 10K iterations in total. Adam optimizer [15] is employed to
minimize the negative log-likelihood loss of Eq. (16). During inference, we
use the greedy decoding strategy to generate captions. More details are shown
in the supplementary material.

Table 1: Comparison with SOTA methods on the Spot-the-Diff dataset.  Model | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE
---|---|---|---|---|---
M-VAM [27] (ECCV 2020) | 10.1 | 12.4 | 31.3 | 38.1 | 14.0
DUDA+TIRG [9] (CVPR 2021) | 8.1 | 12.5 | 29.9 | 34.5 | -
R3Net+SSP [38] (EMNLP 2021) | - | 13.1 | 32.6 | 36.6 | 18.8
VACC [14] (ICCV 2021) | 9.7 | 12.6 | 32.1 | 41.5 | -
MCCFormers-D [26] (ICCV 2021) | 10.0 | 12.4 | - | 43.1 | 18.3
IFDC [11] (TMM 2022) | 8.7 | 11.7 | 30.2 | 37.0 | -
I3N-TD [48] (TMM 2023) | 10.3 | 13.0 | 31.5 | 42.7 | 18.6
VARD-Trans [32] (TIP 2023) | - | 12.5 | 29.3 | 30.3 | 17.3
SCORER+CBR [36] (ICCV 2023) | 10.2 | 13.2 | - | 38.9 | 18.4
SMART [35] (TPAMI 2024) | - | 13.5 | 31.6 | 39.4 | 19.0
DIRL+CCR (Ours) | 10.3 | 13.8 | 32.8 | 40.9 | 19.9

###  4.3 Performance Comparison

Results on the Spot-the-Diff Dataset. This dataset contains image pairs from
surveillance cameras with underlying illumination changes. We compare DIRL+CCR
with the state-of-the-art methods: M-VAM [27], DUDA+TIRG [9], R3Net+SSP [38],
VACC [14], MCCFormers-D [26], IFDC [11], I3N-TD [48], VARD-Trans [32],
SCORER+CBR [36], and SMART [35].

The results are shown in Table 1. Our DIRL+CCR obtains the best results on
most metrics. Compared with the latest works SCORER+CBR and SMART, DIRL+CCR
surpasses them on all metrics, especially with increases of 8.2% and 4.7% on
SPICE. As the image pairs on this dataset are from surveillance cameras, most
objects have no good postures, which makes the model difficult to locate and
describe changed objects. In this situation, our DIRL+CCR also achieves an
encouraging performance, which shows its robustness.

Table 2: Comparison with SOTA methods on the CLEVR-Change dataset.  Model | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE
---|---|---|---|---|---
DUDA [25] (ICCV 2019) | 42.9 | 29.7 | - | 94.6 | 19.9
DUDA+TIRG [9] (CVPR 2021) | 49.9 | 34.3 | 65.4 | 101.3 | 27.9
MCCFormers-D [26] (CVPR 2021) | 53.3 | 37.1 | 70.8 | 119.1 | 30.4
R3Net+SSP [38] (EMNLP 2021) | 52.7 | 36.2 | 69.8 | 116.6 | 30.3
IFDC [11] (TMM 2022) | 47.2 | 29.3 | 63.7 | 105.4 | -
I3N [48] (TMM 2023) | 53.1 | 37.0 | 70.8 | 117.0 | 32.1
NCT [34] (TMM 2023) | 53.1 | 36.5 | 70.7 | 118.4 | 30.9
VARD-Trans [32] (TIP 2023) | 53.6 | 36.7 | 71.0 | 119.1 | 30.5
SCORER+CBR [36] (ICCV 2023) | 54.4 | 37.6 | 71.7 | 122.4 | 31.6
SMART [35] (TPAMI 2024) | 54.3 | 37.4 | 71.8 | 123.6 | 32.0
DIRL+CCR (Ours) | 54.6 | 38.1 | 71.9 | 123.6 | 31.8
Table 3: Evaluation on CLEVR-Change with varied change categories by METEOR. | METEOR |
---|---|---
Model | Color | Texture | Add | Drop | Move
DUDA [25] (ICCV 2019) | 32.8 | 27.3 | 33.4 | 31.4 | 23.5
DUDA+TIRG [9] (CVPR 2021) | 36.1 | 30.4 | 37.8 | 36.7 | 27.0
R3Net+SSP [38] (EMNLP 2021) | 38.9 | 35.5 | 38.0 | 37.5 | 30.9
IFDC [11] (TMM 2022) | 33.1 | 27.9 | 36.2 | 31.4 | 31.2
I3N [48] (TMM 2023) | 39.9 | 36.7 | 39.9 | 38.1 | 30.6
NCT [34] (TMM 2023) | 39.1 | 36.3 | 39.0 | 37.2 | 30.5
SMART [35] (TPAMI 2024) | 40.2 | 37.8 | 39.3 | 38.1 | 31.5
DIRL+CCR (Ours) | 40.7 | 38.2 | 40.0 | 37.9 | 33.5
Table 4: Comparison with SOTA methods on the CLEVR-DC dataset. Model | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE
---|---|---|---|---|---
DUDA [25] (ICCV 2019) | 40.3 | 27.1 | - | 56.7 | 16.1
M-VAM [27] (ECCV 2020) | 40.9 | 27.1 | - | 60.1 | 15.8
VACC [14] (ICCV 2021) | 45.0 | 29.3 | - | 71.7 | 17.6
NCT [34] (TMM 2023) | 47.5 | 32.5 | 65.1 | 76.9 | 15.6
VARD-Trans [32] (TIP 2023) | 48.3 | 32.4 | - | 77.6 | 15.4
SCORER+CBR [36] (ICCV 2023) | 49.4 | 33.4 | 66.1 | 83.7 | 16.2
DIRL+CCR (Ours) | 51.4 | 32.3 | 66.3 | 84.1 | 16.8

Results on the CLEVR-Change Dataset. This dataset contains basic geometric
objects and moderate distractors. We evaluate DIRL+CCR under the setting of
semantic change & distractors and detailed change categories. The comparison
state-of-the-art methods are: DUDA [25], DUDA+TIRG [9], R3Net+SSP [38],
MCCFormers-D [26], IFDC [11], I3N [48], VARD-Trans [32], SCORER+CBR [36], and
SMART [35].

The results are shown in Table 2-3. In the both tables, our DIRL+CCR gains the
superior results under the setting of semantic change & distractors, and
obtains best results on most change categories. The recent work NCT [34] is a
match-based method under the architecture of transformer, which directly
correlates two image representations to extract the shared features and then
removes them to compute the difference features. We find that DIRL+CCR
outperforms it on all metrics. This comparison validates the effectiveness of
performing distractors-immune representation learning before change
localization, and cross-modal alignment regularizing during caption
generation.

Results on the CLEVR-DC Dataset. The experiment is conducted on CLEVR-DC with
extreme distractors. We compare with the following state-of-the-art methods:
DUDA [25], M-VAM [27], VACC [14], NCT [34], VARD-Trans [32], and SCORER+CBR
[36]. The results are shown in Table 4.

We note that the overall performance of our DIRL+CCR is better. Specifically,
DIRL+CCR obtains the best results on BLEU-4, ROUGE-L, and CIDEr metrics, while
achieving comparable performance against the state-of-the-art methods on the
other metrics. Especially, DIRL+CCR obtains relative improvement of 8.2%, 6.4%
and 4.0% on BLEU-4 against the recent works NCT, VARD-Trans and SCORER+CBR,
respectively. The results validate the robustness of our method under the
disturbance of drastic distractors.

Table 5: Comparison with the SOTA methods on Image Editing Request. “*” indicates that the model is trained by pre-training and fine-tuning strategy. Model | BLEU-4 | METEOR | ROUGE-L | CIDEr
---|---|---|---|---
VIXEN-QFormer* [3] (AAAI 2024) | 7.9 | 14.4 | 33.5 | 35.4
VIXEN-CLIP* [3] (AAAI 2024) | 8.6 | 15.4 | 42.5 | 38.1
DUDA [25] (ICCV 2019) | 6.5 | 12.4 | 37.3 | 22.8
Dyn rel-att [30] (ACL 2019) | 6.7 | 12.8 | 37.5 | 26.4
BDLSCR [46] (IJIS 2022) | 6.9 | 14.6 | 38.5 | 27.7
NCT [34] (TMM 2023) | 8.1 | 15.0 | 38.8 | 34.2
VARD-Trans [32] (TIP 2023) | 10.0 | 14.8 | 39.0 | 35.7
SCORER+CBR [36] (ICCV 2023) | 10.0 | 15.0 | 39.6 | 33.4
SMART [35] (TPAMI 2024) | 10.5 | 15.2 | 39.1 | 37.8
DIRL+CCR (Ours) | 10.9 | 15.0 | 41.0 | 34.1

Results on the Image Editing Request Dataset. To further validate the
generalization, we conduct the experiment on the scenario of image editing.
The comparison methods are DUDA [25], Dynamic rel-att [30], BDLSCR [46], NCT
[34], VARD-Trans [32], SCORER+CBR [36], and SMART [35]. Besides, we compare
our method with the latest work VIXEN [3]. VIXEN is first pre-trained on a
large-scale image editing dataset (which has not been released yet) and fine-
tuned on the Image Editing Request dataset.

The results are shown in Table 5. Compared with the end-to-end training
methods, our DIRL+CCR achieves the best results on two metrics and is on par
with the SOTA methods on other two metrics. Compared with VIXEN, our DIRL+CCR
also achieves a competitive performance. The comparison results validate the
effectiveness and generalization of our method.

Performance Analysis. In brief, compared to the state-of-the-art methods, our
method achieves superior results in different scenarios. This benefits from
that 1) DIRL can make the paired image representations non-perturbational and
discriminative under distractors. This helps learn robust difference features
for caption generation. 2) CCR regularizes the cross-modal alignment by
maximizing the contrastive alignment between the features of generated words
and attended difference, thus helping improve the quality of yielded captions.

Table 6: Ablation study about DIRL and CCR on the CLEVR-DC dataset. Ablation | DIRL | CCR | BLEU-4 | ROUGE-L | CIDEr | SPICE
---|---|---|---|---|---|---
Transformer | $\times$ | $\times$ | 48.9 | 65.6 | 79.6 | 15.7
Transformer | $\checkmark$ | $\times$ | 50.5 | 65.8 | 81.8 | 16.2
Transformer | $\times$ | $\checkmark$ | 49.3 | 65.5 | 82.7 | 16.4
Transformer | $\checkmark$ | $\checkmark$ | 51.4 | 66.3 | 84.1 | 16.8
Table 7: Ablation study about DIRL on the CLEVR-DC dataset. Ablation | non-perturbation | discrimination | BLEU-4 | ROUGE-L | CIDEr | SPICE
---|---|---|---|---|---|---
Transformer | $\times$ | $\times$ | 48.9 | 65.6 | 79.6 | 15.7
DIRL | $\checkmark$ | $\times$ | 50.7 | 65.2 | 79.3 | 16.2
DIRL | $\times$ | $\checkmark$ | 50.1 | 65.1 | 79.5 | 16.2
DIRL | $\checkmark$ | $\checkmark$ | 50.5 | 65.8 | 81.8 | 16.2
Figure 3: Visualization of change localization and captioning results of
DIRL+CCR and Transformer. GT is short for ground-truth and changed objects are
shown in red boxes.

###  4.4 Ablation Study

Ablation study of each module. To figure out the contribution of the proposed
DIRL and CCR, we conduct the ablation study on the CLEVR-DC dataset (with
extreme distractors). The baseline model is a vanilla Transformer. It directly
interacts two image representations to extract share features, which are then
removed from image representations to attain the difference features for
caption generation. The results shown in Table 6.

When we augment the vanilla Transformer with DIRL, it yields a performance
boost. This validates DIRL can learn the two image representations that are
non-perturbational and discriminative under the extreme distractors. As such,
the model can better mine the shared features to learn reliable difference
features. Besides, the performance of vanilla Transformer is enhanced by
equipping it with CCR, which verifies that CCR can regularize the cross-modal
alignment. By using both DIRL and CCR, the vanilla Transformer achieves best
performance, in particular with the improvements of 5.7% and 7.0% on CIDEr and
SPICE metrics. This shows that each module not only plays its unique role, but
also supplements each other. In Fig. 3, we visualize the change localization
and captioning results of DIRL+CCR and vanilla Transformer under distractors,
where the unchanged objects appear obviously pseudo changes about scale and
location. The visualization indicates that with the help of DIRL and CCR, the
model can pinpoint and describe the actually changed object.

Ablation study of DIRL. DIRL aims to learn two stable image representations
from the perspectives of non-perturbation and discrimination, where the former
is achieved by the first term of Eq. (10); the latter is achieved by the
second term. To validate our claim, we conduct the ablative study for non-
perturbation and discrimination, respectively. The results are shown in Table
7. The performance of DIRL with either characteristic does not obtain much
gains against the vanilla Transformer. Our conjecture is that the influence of
distractors is not well solved. By jointly modeling the non-perturbation and
discrimination, the performance of DIRL is significantly improved, which shows
that each characteristic is key to handle distractors. That is, only if the
model learns both, can it make the paired image representations stable under
distractors. As such, the model can capture the reliable difference features
for caption generation.

Figure 4: Visualization of captioning performance under varied viewpoint
changes.

Figure 5: The effects of two trade-off parameters of $\lambda_{d}$ and
$\lambda_{c}$ on CLEVR-DC.

###  4.5 Measuring Captioning Performance under Distractors

To verify whether our method is immune to distractors, we measure the model’s
captioning performance under varied viewpoint changes. The varied viewpoints
are computed by the IoU of the bounding boxes of the objects of two images,
where the more the camera changes its position, the less the bounding boxes
overlap. That is, lower IoU means higher difficulty. The results are shown in
Fig. 5, where the compared models are Subtraction (direct subtraction between
two image representations), vanilla Transformer, DIRL and DIRL+CCR. We find
that the performance of Subtraction is the lowest, showing that direct
subtraction generalizes poorly to distractors. DIRL apparently surpasses
Transformer, validating it does make the representations of image pair immune
to distractors. Further, the performance of DIRL further boosts by augmenting
it with CCR, which demonstrates that CCR plays a role in enhancing cross-modal
alignment and thus improves the captioning quality under distractors.

###  4.6 Study on the Trade-off Parameters $\lambda_{d}$ and $\lambda_{c}$

We discuss the effects of two trade-off parameters $\lambda_{d}$ and
$\lambda_{c}$ in Eq.(16). Fig. 5 shows the results of DIRL and DIRL+CCR with
different $\lambda_{d}$ and $\lambda_{c}$ on the CLEVR-DC dataset. We first
discuss $\lambda_{d}$. As its value increases or decreases, the performance of
DIRL changes. Based the results, we set $\lambda_{d}$ as 0.03. Then, we fix
$\lambda_{d}$ to discuss $\lambda_{c}$ and set it as $0.05$. Similarly, we
further discuss $\lambda_{d}$ and $\lambda_{c}$ on the other three datasets,
and set them as 0.5 and 0.3 on the CLEVR-Change dataset; 0.5 and 0.004 on the
Spot-the-Diff dataset; 0.001 and 0.05 on the Image Editing Request dataset.

Figure 6: Qualitative analysis on four datasets. We visualize the ground-truth
captions (GT), generated captions and change localization results of
SCORER+CBR [36] and our DIRL+CCR. The ground-truth changes are shown in red
boxes.

###  4.7 Qualitative Analysis

We conduct qualitative analysis about change localization and caption
generation on the four datasets, as shown in Fig. 6 (a)-(d). The compared
method is SCORER+CBR [36], which first obtains view-invariant representations
and then interacts both to compute the difference features, as well as
improving captioning quality by cross-model backward reasoning. In these
examples, the change localization maps of SCORER+CBR are relatively scattered
and cannot focus on genuine changes. Hence, it fails to generate the correct
sentences to describe these changes. By contrast, our DIRL+CCR successfully
recognizes and captions the changes. This superiority benefits from making two
image representations non-perturbational and discriminative under distractors.
As such, our method can better pinpoint the actually changed object between
them. During caption generation, CCR further regularizes cross-modal
alignment, so as to make the target words generated based on the most related
difference features. More qualitative examples are shown in the supplementary
material.

##  5 Conclusion

In this paper, we propose the DIRL to learn a pair of non-perturbational and
discriminative image representations under distractors, by correlating their
corresponding channels and decorrelating different channels. As such, the
model can sufficiently mine their shared features to learn the robust
difference features for caption generation. Further, we design the CCR to
regularize the cross-modal alignment by maximizing the contrastive alignment
between the attended difference features and generated words, which helps
improve captioning performance. Extensive experiments show that our method
yields state-of-the-art results on the four public datasets with different
change scenes.

## Acknowledgements

This work was supported in part by National Natural Science Foundation of
China: 62322211, 61931008, 62236008, 62336008, U21B2038, 62225207, Fundamental
Research Funds for the Central Universities (E2ET1104), “Pionee” and “Leading
Goose” R&D Program of Zhejiang Province (2024C01023).

## References

  * [1] Anderson, P., Fernando, B., Johnson, M., Gould, S.: Spice: Semantic propositional image caption evaluation. In: ECCV. pp. 382–398. Springer (2016)
  * [2] Banerjee, S., Lavie, A.: Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. pp. 65–72 (2005)
  * [3] Black, A., Shi, J., Fai, Y., Bui, T., Collomosse, J.: Vixen: Visual text comparison network for image difference captioning. In: AAAI (2024)
  * [4] Chen, J., Li, L., Su, L., Zha, Z.j., Huang, Q.: Prompt-enhanced multiple instance learning for weakly supervised video anomaly detection. In: CVPR. pp. 18319–18329 (2024)
  * [5] Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollár, P., Zitnick, C.L.: Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 (2015)
  * [6] Cho, J., Yoon, S., Kale, A., Dernoncourt, F., Bui, T., Bansal, M.: Fine-grained image captioning with clip reward. In: Findings of NAACL. pp. 517–527 (2022)
  * [7] Guo, Z., Wang, T.J., Laaksonen, J.: Clip4idc: Clip for image difference captioning. In: AACL. pp. 33–42 (2022)
  * [8] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)
  * [9] Hosseinzadeh, M., Wang, Y.: Image change captioning by learning from an auxiliary task. In: CVPR. pp. 2725–2734 (2021)
  * [10] Hoxha, G., Chouaf, S., Melgani, F., Smara, Y.: Change captioning: A new paradigm for multitemporal remote sensing image analysis. IEEE Transactions on Geoscience and Remote Sensing 60, 1–14 (2022)
  * [11] Huang, Q., Liang, Y., Wei, J., Cai, Y., Liang, H., Leung, H.f., Li, Q.: Image difference captioning with instance-level fine-grained feature representation. IEEE Transactions on Multimedia 24, 2004–2017 (2022)
  * [12] Islam, M.M., Ho, N., Yang, X., Nagarajan, T., Torresani, L., Bertasius, G.: Video recap: Recursive captioning of hour-long videos. In: CVPR (2024)
  * [13] Jhamtani, H., Berg-Kirkpatrick, T.: Learning to describe differences between pairs of similar images. In: EMNLP. pp. 4024–4034 (2018)
  * [14] Kim, H., Kim, J., Lee, H., Park, H., Kim, G.: Agnostic change captioning with cycle consistency. In: ICCV. pp. 2095–2104 (2021)
  * [15] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [16] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: ICCV. pp. 4015–4026 (2023)
  * [17] Li, L., Gao, X., Deng, J., Tu, Y., Zha, Z.J., Huang, Q.: Long short-term relation transformer with global gating for video captioning. IEEE Transactions on Image Processing 31, 2726–2738 (2022)
  * [18] Li, M., Lin, B., Chen, Z., Lin, H., Liang, X., Chang, X.: Dynamic graph enhanced contrastive learning for chest x-ray report generation. In: CVPR. pp. 3334–3343 (2023)
  * [19] Liao, Z., Huang, Q., Liang, Y., Fu, M., Cai, Y., Li, Q.: Scene graph with 3d information for change captioning. In: ACM MM. pp. 5074–5082 (2021)
  * [20] Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out. pp. 74–81 (2004)
  * [21] Liu, C., Zhao, R., Chen, H., Zou, Z., Shi, Z.: Remote sensing image change captioning with dual-branch transformers: A new method and a large scale dataset. IEEE Transactions on Geoscience and Remote Sensing 60, 1–20 (2022)
  * [22] Liu, X., Li, L., Wang, S., Zha, Z.J., Li, Z., Tian, Q., Huang, Q.: Entity-enhanced adaptive reconstruction network for weakly supervised referring expression grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(3), 3003–3018 (2023)
  * [23] Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
  * [24] Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: ACL. pp. 311–318 (2002)
  * [25] Park, D.H., Darrell, T., Rohrbach, A.: Robust change captioning. In: ICCV. pp. 4624–4633 (2019)
  * [26] Qiu, Y., Yamamoto, S., Nakashima, K., Suzuki, R., Iwata, K., Kataoka, H., Satoh, Y.: Describing and localizing multiple changes with transformers. In: ICCV. pp. 1971–1980 (2021)
  * [27] Shi, X., Yang, X., Gu, J., Joty, S., Cai, J.: Finding it at another side: A viewpoint-adapted matching encoder for change captioning. In: ECCV. pp. 574–590 (2020)
  * [28] Sun, Y., Qiu, Y., Khan, M., Matsuzawa, F., Iwata, K.: The stvchrono dataset: Towards continuous change recognition in time. In: CVPR. pp. 14111–14120 (2024)
  * [29] Sun, Y., Li, L., Yao, T., Lu, T., Zheng, B., Yan, C., Zhang, H., Bao, Y., Ding, G., Slabaugh, G.: Bidirectional difference locating and semantic consistency reasoning for change captioning. International Journal of Intelligent Systems 37(5), 2969–2987 (2022)
  * [30] Tan, H., Dernoncourt, F., Lin, Z., Bui, T., Bansal, M.: Expressing visual relationships via language. In: ACL. pp. 1873–1883 (2019)
  * [31] Tang, W., Li, L., Liu, X., Jin, L., Tang, J., Li, Z.: Context disentangling and prototype inheriting for robust visual grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence 46(5), 3213–3229 (2024)
  * [32] Tu, Y., Li, L., Su, L., Du, J., Lu, K., Huang, Q.: Viewpoint-adaptive representation disentanglement network for change captioning. IEEE Transactions on Image Processing 32, 2620–2635 (2023)
  * [33] Tu, Y., Li, L., Su, L., Gao, S., Yan, C., Zha, Z.J., Yu, Z., Huang, Q.: I2transformer: Intra-and inter-relation embedding transformer for tv show captioning. IEEE Transactions on Image Processing 31, 3565–3577 (2022)
  * [34] Tu, Y., Li, L., Su, L., Lu, K., Huang, Q.: Neighborhood contrastive transformer for change captioning. IEEE Transactions on Multimedia 25, 9518–9529 (2023)
  * [35] Tu, Y., Li, L., Su, L., Zha, Z.J., Huang, Q.: Smart: Syntax-calibrated multi-aspect relation transformer for change captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence 46(7), 4926–4943 (2024)
  * [36] Tu, Y., Li, L., Su, L., Zha, Z.J., Yan, C., Huang, Q.: Self-supervised cross-view representation reconstruction for change captioning. In: ICCV. pp. 2805–2815 (2023)
  * [37] Tu, Y., Li, L., Su, L., Zha, Z.J., Yan, C., Huang, Q.: Context-aware difference distilling for multi-change captioning. In: ACL (2024)
  * [38] Tu, Y., Li, L., Yan, C., Gao, S., Yu, Z.: R^3Net:relation-embedded representation reconstruction network for change captioning. In: EMNLP. pp. 9319–9329 (2021)
  * [39] Tu, Y., Yao, T., Li, L., Lou, J., Gao, S., Yu, Z., Yan, C.: Semantic relation-aware difference representation learning for change captioning. In: Findings of ACL. pp. 63–73 (2021)
  * [40] Tu, Y., Zhou, C., Guo, J., Li, H., Gao, S., Yu, Z.: Relation-aware attention for video captioning via graph learning. Pattern Recognition 136, 109204 (2023)
  * [41] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NeurIPS. pp. 5998–6008 (2017)
  * [42] Vedantam, R., Lawrence Zitnick, C., Parikh, D.: Cider: Consensus-based image description evaluation. In: CVPR. pp. 4566–4575 (2015)
  * [43] Wang, Q., Zhang, Y., Zheng, Y., Pan, P., Hua, X.S.: Disentangled representation learning for text-video retrieval. arXiv preprint arXiv:2203.07111 (2022)
  * [44] Xiao, J., Li, L., Lv, H., Wang, S., Huang, Q.: R&b: Region and boundary aware zero-shot grounded text-to-image generation. ICLR (2024)
  * [45] Yang, C.F., Tsai, Y.H.H., Fan, W.C., Salakhutdinov, R.R., Morency, L.P., Wang, F.: Paraphrasing is all you need for novel object captioning. NeurIPS 35, 6492–6504 (2022)
  * [46] Yao, L., Wang, W., Jin, Q.: Image difference captioning with pre-training and contrastive learning. In: AAAI (2022)
  * [47] Yue, S., Tu, Y., Li, L., Gao, S., Yu, Z.: Multi-grained representation aggregating transformer with gating cycle for change captioning. ACM Trans. Multimedia Comput. Commun. Appl. (2024)
  * [48] Yue, S., Tu, Y., Li, L., Yang, Y., Gao, S., Yu, Z.: I3n: Intra- and inter-representation interaction network for change captioning. IEEE Transactions on Multimedia 25, 8828–8841 (2023)
  * [49] Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow twins: Self-supervised learning via redundancy reduction. In: ICML. pp. 12310–12320 (2021)
  * [50] Zhang, D., Zhang, H., Tang, J., Hua, X.S., Sun, Q.: Causal intervention for weakly-supervised semantic segmentation. NeurIPS 33, 655–666 (2020)

##  6 Appendix

###  6.1 Experiments

####  6.1.1 Implementation Details

For a fair comparison, we follow the state-of-the-art methods [36, 35] to
utilize a pre-trained ResNet-101 [8] model to obtain the features of a pair of
images, with the dimension of 1024 $\times$ 14 $\times$ 14\. We project them
into a lower dimension of 512. The hidden size of the model and word embedding
size are set to 512 and 300, separately. Temperature $\tau$ in Eq. (14) of
main paper is set to 0.5.

In the training phase, the batch sizes and learning rates of our method on the
four datasets are shown in Table 8. We use Adam optimizer [15] to minimize the
negative log-likelihood loss of Eq. (16) of main paper. In the inference
phase, we use the greedy decoding strategy to generate captions. Both training
and inference are implemented with PyTorch on an RTX 3090 GPU. The used
training resources on the all datasets are shown in Table 9. We find that our
method does not require much training time and GPU memory, so it can be easily
reproduced by the researchers. To facilitate future research, the code is
publicly available at https://github.com/tuyunbin/DIRL.

Table 8: The training parameters of our method on the four datasets. | batch size | learning rate
---|---|---
CLEVR-Change | 128 | 2 $\times$ $10^{-4}$
CLEVR-DC | 128 | 2 $\times$ $10^{-4}$
Spot-the-Diff | 64 | 1 $\times$ $10^{-4}$
Image Editing Request | 16 | 1 $\times$ $10^{-4}$
Table 9: The used training time and GPU memory on the four datasets. | Training Time | GPU Memory
---|---|---
CLEVR-Change | 150 minutes | 14 GB
CLEVR-DC | 90 minutes | 8.4 GB
Spot-the-Diff | 25 minutes | 6.5 GB
Image Editing Request | 10 minutes | 3.2 GB
Table 10: Effect of trade-off parameter $\alpha$ in DIRL on the CLEVR-DC dataset. $\alpha$ | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE
---|---|---|---|---|---
0.001 | 50.6 | 32.3 | 65.7 | 81.8 | 16.1
0.002 | 50.4 | 32.5 | 65.9 | 81.2 | 16.1
0.003 | 50.5 | 32.5 | 65.8 | 81.8 | 16.2
0.004 | 50.4 | 32.3 | 65.7 | 80.5 | 16.3
0.005 | 50.1 | 31.9 | 65.4 | 79.7 | 16.2
0.006 | 50.5 | 31.5 | 64.9 | 79.8 | 15.8

####  6.1.2 Study on the Trade-off Parameter $\alpha$

In this section, we discuss the effect of trade-off parameter $\alpha$ in Eq.
(4) of main paper. As mentioned in Sec. 3.2 Distractors-Immune Representation
Learning, $\alpha$ is a trade-off parameter to balance the importance between
the two terms. The first term equates the diagonal of $\mathcal{C}$ to one,
i.e., the corresponding feature channels of two image representations will be
correlated and thus have similar semantics under distractors. The second term
equates the off diagonal of $\mathcal{C}$ to zero, i.e., the different
channels will be decorrelated. This enhances the discrimination of each image
representation. Both terms are key to handle the influence of distractors.
Since the CLEVR-DC dataset is with extreme distractors and more challenging,
we conduct experiment on this dataset. In Table 10, we find that the
captioning results are close when setting the value of $\alpha$ from 0.001 to
0.006, and the model’s overall performance is better under the value of 0.003.
This shows that the proposed DIRL is robust. Empirically, we set the value of
$\alpha$ as 0.003 on the four datasets.

Table 11: Ablation of DIRL with/without the MLP on the CLEVR-DC dataset.  Ablation | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE
---|---|---|---|---|---
DIRL w/ MLP | 51.4 | 32.3 | 66.3 | 84.1 | 16.8
DIRL w/o MLP | 48.0 | 32.0 | 65.2 | 78.5 | 15.7

####  6.1.3 Study on the MLP Function in DIRL

In Eq.(1) of the main paper, a convolution function transforms the features of
two images into a low dimension, and then the trainable position encodings are
integrated into the transformed features along their height and width.
Further, the MLP in Eq.(2) projects the two position-embedded features into a
shared embedding space. To study the effect of the MLP, we carry out the
ablation study of DIRL with/without MLP on the CLEVR-DC dataset, which is show
in Table 11. It is noted that DIRL with MLP is much better than it without
MLP, which shows that adding the MLP helps DIRL learn a pair of distractors-
immune representations in terms of semantics and position.

####  6.1.4 Comparison between DIRL and Static Methods

The proposed DIRL aims to learn a pair of distractors-immune representations
by correlating the corresponding feature channels and decorrelating different
ones. In fact, there are some simpler static methods such as PCA or ZCA
whitening to remove the degree of correlation. Here, we conduct the experiment
to show performance comparison of cross-channel decorrelation among the
transformer-based model with PCA/ZCA whitening and our DIRL.

Table 12: Performance comparison among Transformer-based model with PCA/ZCA whitening and our DIRL.  Ablative Variants | BLEU-4 | ROUGE-L | CIDEr | SPICE
---|---|---|---|---
Transformer | 48.9 | 65.6 | 79.6 | 15.7
Transformer w/ PCA whitening | 40.6 | 57.9 | 22.0 | 10.0
Transformer w/ ZCA whitening | 38.6 | 57.3 | 33.3 | 13.1
Transformer w/ DIRL | 50.5 | 65.8 | 81.8 | 16.2

The two static methods are used as data preprocessing strategies before model
training, while the proposed DIRL is jointly trained with the model. The
comparison results are shown in Table 12. We can find that the model with DIRL
outperforms the others by a large margin, indicating that joint training
strategy (DIRL) does help the model learn two stable image features under
distractors.

Figure 7: Visualization of shared objects matching between two images on the
CLEVR-Change and CLEVR-DC with distractors to varying degrees. For each
example, we visualize matching results by the classic match-based method
MCCFormers-D [26] and the proposed DIRL+CCR. The changed objects are shown in
the red boxes. Besides, we visualize the ground-truth caption (GT), and the
generated captions by MCCFormers-D and our DIRL+CCR. The correct words are in
green color, while incorrect words are in red color.  Figure 8: Visualization
of shared objects matching between two images on the Image Editing Request
(IER) and Spot-the-Diff datasets. For each example, we visualize matching
results by the classic match-based method MCCFormers-D [26] and the proposed
DIRL+CCR. The changed objects are shown in the red boxes. Besides, we
visualize the ground-truth caption (GT), and the generated captions by
MCCFormers-D and our DIRL+CCR. The correct words are in green color, while
incorrect words are in red color.

####  6.1.5 Qualitative Analysis

In this supplementary material, we will show more qualitative examples on the
CLEVR-Change, CLEVR-DC, Spot-the-Diff, and Image Editing Request datasets,
which are shown in Fig. 7-11. In Fig. 7-8, on the four datasets, we visualize
shared objects matching between two images, which are yielded by the classic
match-based method MCCFormers-D [26] and the proposed DIRL+CCR. We can find
that MCCFormers-D is unable to sufficiently or correctly align the shared
objects between two images. By contrast, the proposed DIRL+CCR can better
match the shared objects. These examples validate that the proposed DIRL can
make the representations of image pair non-perturbational and discriminative
under the distractors to varying degrees. Based on the two distractors-immune
image representations, the model can better interact and mine their shared
features.

To intuitively validate the change localization and caption capabilities of
our method, we visualize the generated captions by DIRL+CCR under different
change types on the four datasets. Meanwhile, we visualize the change
localization results that are obtained from the cross-attention maps between
the difference features and generated words in the decoding process. These are
shown in Fig. 9-11. When the attention score is higher, the region is
brighter. We observe that the proposed DIRL+CCR can accurately localize and
describe the actually changed objects under different scenarios. This
superiority mainly benefits from the facts that 1) DIRL is able to learn two
distractors-immune image representations for matching the shared objects, so
as to learn the reliable difference features for caption generation; 2) CCR is
capable of regularizing the cross-modal alignment by maximizing the
contrastive alignment between the generated words and attended difference
features, so as to improve the quality of yielded captions.

### Limitation

Fig. 12 shows a failure case that derives from our DIRL+CCR. This image pair
is from the surveillance cameras and with underlying distractors (illumination
change), where there are three people newly appearing in the “after” image.
DIRL+CCR can accurately localize the region containing the added people and
describe the change type, which benefit from the proposed DIRL and CCR.
However, we note that the generated sentence wrongly describes the number of
added people. Our conjecture is that under surveillance cameras, the changed
objects are commonly small. Further, the distance of the left two people is
very close, and the color of one man’s coat is similar to the other’s pants.
In this situation, the model risks recognizing two people as one person, so as
to generate the incorrect result. In our opinion, a possible solution is to
leverage finer-level visual modality for the representation of such small
objects, _e.g.,_ image segmentation features [50, 16].

Figure 9: Four cases from the CLEVR-Change dataset. We visualize the ground-
truth caption (GT), and the captions yielded by our DIRL+CCR. We also
visualize its change localization results. The ground-truth changed objects
are shown in red boxes. Figure 10: Four cases from the CLEVR-DC dataset. We
visualize the ground-truth caption (GT), and the captions yielded by our
DIRL+CCR. We also visualize its change localization results. The ground-truth
changed objects are shown in red boxes. Figure 11: Four cases from the Image
Editing Request (IER) and Spot-the-Diff datasets. We visualize the ground-
truth caption (GT), and the captions yielded by our DIRL+CCR. We also
visualize its change localization results. The ground-truth changed objects
are shown in red boxes. Figure 12: The failure case that derives from the
proposed DIRL+CCR.


# On some weighted maximal operators of partial sums of Walsh-Fourier series
in the space $H_{1}$

Davit Baramidze  D. Baramidze, The University of Georgia, School of science
and technology, 77a Merab Kostava St, Tbilisi 0128, Georgia and Department of
Computer Science and Computational Engineering, UiT - The Arctic University of
Norway, P.O. Box 385, N-8505, Narvik, Norway<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

In the first part of this paper we describe the status of the art of this
subject. In the second part we present and motivate some new results. Indeed,
we introduce some new weighted maximal operators of the partial sums of the
Walsh-Fourier series. We prove that for some "optimal" weights these new
operators indeed are bounded from the martingale Hardy space $H_{1}(G)$ to the
space $\text{weak}-L_{1}(G),$ but is not bounded from $H_{1}(G)$ to the space
$L_{1}(G).$

The research was supported by Shota Rustaveli National Science Foundation
grant no. PHDF-21-1702.

2010 Mathematics Subject Classification. 42C10.

Key words and phrases: Walsh-Fourier series, partial sums, martingale Hardy
space, maximal operators, weighted maximal operators.

##  1\. Introduction and status of the art

It is well-known that the Walsh system does not form a basis in the space
$L_{1}$ (see e.g. [2]). Moreover, there exists a function in the dyadic Hardy
space $H_{1}(G),$ such that the partial sums of $f$ are not bounded in the
$L_{1}$-norm. Uniform and pointwise convergence and some approximation
properties of partial sums in $L_{1}(G)$ norms were investigated by Avdispahić
and Memić [1], Gát, Goginava and Tkebuchava [15, 16], Nagy [21, 22], Onneweer
[23] , Paley [24], and Persson, Schipp, Tephnadze and Weisz [27]. Fine [12]
obtained sufficient conditions for the uniform convergence which are
completely analogous to the Dini-Lipschits conditions. Gulicev [18] estimated
the rate of uniform convergence of a Walsh-Fourier series by using Lebesgue
constants and modulus of continuity. These problems for Vilenkin groups were
investigated by Blatota, Nagy, Persson and Tephnadze [10] (see also [7, 6, 8,
9, 25]), Fridli [13], Gát [14], Simon [29] and Tephnadze [20, 34, 35].

In the study of convergence of subsequences of partial sums and their
restricted maximal operators on the martingale Hardy spaces $H_{p}$ for
$0<p\leq 1,$ the central role is played by the fact that any natural number
$n\in\mathbb{N}$ can be uniquely expressed as
$n=\sum_{k=0}^{\infty}n_{j}2^{j},\ \ n_{j}\in Z_{2}~{}(j\in\mathbb{N}),$ where
only a finite numbers of $n_{j}$ differ from zero and their important
characters $\left[n\right],$ $\left|n\right|,$ $\rho\left(n\right)$ and $V(n)$
are defined by

$\left[n\right]:=\min\\{j\in\mathbb{N},n_{j}\neq 0\\},\ \ \ \ \
\left|n\right|:=\max\\{j\in\mathbb{N},n_{j}\neq 0\\},\ \ \ \ \
\rho\left(n\right)=\left|n\right|-\left[n\right]$

and

$V\left(n\right):=n_{0}+\overset{\infty}{\underset{k=1}{\sum}}\left|n_{k}-n_{k-1}\right|,\text{
\ for \ all \ \ }n\in\mathbb{N}.$

Moreover, every $n\in\mathbb{N}$ can be also represented as
$n=\sum_{i=1}^{r}2^{n_{i}},n_{1}>n_{2}>...n_{r}\geq 0$ and for any
$\\{n_{s_{j}}\\},j=1,2,\ldots,r,$ satisfying
$2^{s}\leq{{n}_{{{s}_{1}}}}\leq{{n}_{{{s}_{2}}}}\leq...\leq{{n}_{{{s}_{r}}}}<{2^{s+1}},\
s\in\mathbb{N},$ we define numbers

(1) $s_{-}:=\min\\{\left[n_{s_{j}}\right]\\},\ \ \
s_{+}:=\max\\{\left|n_{s_{j}}\right|\\}=s,\ \ \
\rho_{s}\left(n_{s_{j}}\right):=s_{+}-s_{-}.$

In particular, (see [11], [19] and [28])

$V\left(n\right)/8\leq\|D_{n}\|_{1}\leq V\left(n\right)$

Hence, for any $F\in L_{1},$ there exists an absolute constant $c$ such that

$\left\|S_{n}F\right\|_{1}\leq c{V\left(n\right)}\left\|F\right\|_{1}.$

Moreover, for any $f\in H_{1}$ (see [32])

$\left\|S_{n}F\right\|_{H_{1}}\leq
c{V\left(n\right)}\left\|F\right\|_{H_{1}}.$

For $0<p<1$ in [30, 31] the weighted maximal operator
$\overset{\sim}{S}^{*,p},$ defined by

(2)
$\overset{\sim}{S}^{*,p}F:=\sup_{n\in\mathbb{N}}\frac{\left|S_{n}F\right|}{\left(n+1\right)^{1/p-1}}$

was investigated and it was proved that the following inequalities hold:

$\left\|\overset{\sim}{S}^{*}F\right\|_{p}\leq c_{p}\left\|F\right\|_{H_{p}}$

Moreover, it was also proved that the rate of the sequence
$\\{\left(n+1\right)^{1/p-1}\\}$ given in the denominator of (2) can not be
improved.

In [32] and [33] it was proved that if $F\in H_{p},$ then there exists an
absolute constant $c_{p},$ depending only on $p,$ such that

$\text{ }\left\|S_{n}F\right\|_{H_{p}}\leq
c_{p}2^{\rho\left(n\right)\left(1/p-1\right)}\left\|F\right\|_{H_{p}}.$

In [5] it was proved that the maximal operator

(3)
$\sup_{n\in\mathbb{N}}\frac{\left|S_{n}F\right|}{2^{\rho\left(n\right)\left(1/p-1\right)}}$

is bounded from $H_{p}$ to $\text{weak}-L_{p}.$ Moreover, if $0<p<1,$
$\left\\{n_{k}:\text{ }k\geq 0\right\\}$ is any increasing sequence of
positive integers such that
$\sup_{k\in\mathbb{N}}\rho\left(n_{k}\right)=\infty$ and
$\Phi:\mathbb{N}_{+}\rightarrow\mathbb{R}$ is any nondecreasing function,
satisfying the condition

$\overline{\underset{k\rightarrow\infty}{\lim}}\frac{2^{\rho\left(n_{k}\right)\left(1/p-1\right)}}{\Phi\left(n_{k}\right)}=\infty,$

then there exists a martingale $F\in H_{p}(G),$ such that

$\underset{k\in\mathbb{N}}{\sup}\left\|\frac{S_{n_{k}}F}{\Phi\left(n_{k}\right)}\right\|_{\text{weak}-L_{p}}=\infty.$

In [5] it was also proved that the weighted maximal operator (3) is not
bounded from $H_{p}(G)$ to the Lebesgue space $L_{p}(G),$ for $0<p<1.$

In [3] it was proved that if $0<p<1,$ $f\in{{H}_{p}},$ $\left\\{n_{k}:k\geq
0\right\\}$ is a sequence of positive integers and $\left\\{n_{s_{i}}:\ 1\leq
i\leq r\right\\}\subset\left\\{n_{k}:\ k\geq 0\right\\}$ are integers such
that
$2^{s}\leq{{n}_{{{s}_{1}}}}\leq{{n}_{{{s}_{2}}}}\leq...\leq{{n}_{{{s}_{r}}}}\leq{2^{s+1}},\
s\in\mathbb{N},$ then the weighted maximal operator
$\widetilde{S}^{\ast,\nabla},$ defined by

$\widetilde{S}^{\ast,\nabla}F:=\sup_{s\in\mathbb{N}}\sup_{2^{s}\leq
n_{s_{i}}<2^{s+1}}\frac{\left|S_{n}F\right|}{2^{\rho_{s}\left(n_{s_{i}}\right)\left(1/p-1\right)}},$

where $\rho_{s}\left(n_{s_{i}}\right)$ are defined by (1), is bounded from the
Hardy space ${{H}_{p}}$ to the Lebesgue space ${{L}_{p}}$. Moreover, if
$0<p<1,$ $\left\\{n_{k}:k\geq 0\right\\}$ is a sequence of positive numbers
and $\left\\{n_{s_{i}}:\ 1\leq i\leq r\right\\}\subset\left\\{n_{k}:\ k\geq
0\right\\}$ is a subsequence satisfying the condition
$2^{s}\leq{{n}_{{{s}_{1}}}}\leq{{n}_{{{s}_{2}}}}\leq...\leq{{n}_{{{s}_{r}}}}\leq{2^{s+1}},\
s\in\mathbb{N},$ then, for any nonnegative, nondecreasing function
$\varphi:\mathbb{N}_{+}\to\mathbb{R}$ satisfying condition

$\sup_{s\in\mathbb{N}}\sup_{2^{s}\leq
n_{s_{i}}<2^{s+1}}\frac{2^{\rho_{s}\left(n_{s_{i}}\right)\left(1/p-1\right)}}{\varphi(n_{s_{i}})}=\infty,$

the maximal operator, defined by

$\sup_{s\in\mathbb{N}}\sup_{2^{s}\leq
n_{s_{i}}<2^{s+1}}\frac{\left|S_{n}F\right|}{\varphi\left(n_{s_{i}}\right)},$

is not bounded from the Hardy space ${{H}_{p}}$ to the Lebesgue space
${{L}_{p}}$.

In [4] it was proved that if $0<p<1$, $f\in{{H}_{p}}\left(G\right)$ and
$\varphi:\mathbb{N}_{+}\to\mathbb{R}_{+}$ be any nonnegative and nondecreasing
function satisfying the condition

$\sum_{n=1}^{\infty}\frac{1}{\varphi^{p}(n)}<c<\infty,$

then, for any sequence $\left\\{n_{k}:k\geq 0\right\\}$ of positive integers,
the weighted maximal operator $\widetilde{S}^{\ast,\nabla},$ defined by

(4)
$\widetilde{S}^{\ast,\nabla}F=\underset{k\in\mathbb{N}}{\sup}\frac{\left|S_{n_{k}}F\right|}{2^{\rho\left(n_{k}\right)\left(1/p-1\right)}\varphi(\rho\left(n_{k}\right))},$

is bounded from the Hardy space ${{H}_{p}}$ to the Lebesgue space ${{L}_{p}}$.
Moreover, for any $0<p<1$ and any sequence $\left\\{n_{k}:k\geq 0\right\\}$ of
positive numbers and $\varphi:\mathbb{N}_{+}\to\mathbb{R}_{+}$ be any
nonnegative and nondecreasing function satisfying the condition

$\sum_{n=1}^{\infty}\frac{1}{\varphi^{p}(n)}=\infty,$

the weighted maximal operator $\widetilde{S}^{\ast,\nabla},$ defined by (4),
is not bounded from the Hardy space ${{H}_{p}}(G)$ to the Lebesgue space
${{L}_{p}}(G)$. Hence, we get that if $0<p<1$ and
$f\in{{H}_{p}}\left(G\right)$, then the weighted maximal operator
$\widetilde{S}^{\ast,\nabla,\varepsilon},$ defined by

$\widetilde{S}^{\ast,\nabla,\varepsilon}F:=\underset{n\in\mathbb{N}}{\sup}\frac{\left|S_{n}F\right|}{2^{\rho\left(n\right)\left(1/p-1\right)}\left(\rho\left(n\right)\log^{1+\varepsilon}\rho\left(n\right)\right)^{1/p}},\
\ \ \varepsilon>0,$

is bounded from the Hardy space ${{H}_{p}}(G)$ to the Lebesgue space
${{L}_{p}}(G)$ for $\varepsilon>0$ and is not bounded from the Hardy space
${{H}_{p}}(G)$ to the Lebesgue space ${{L}_{p}}(G)$ for $\varepsilon=0.$

In this paper we prove that some new weighted maximal operators of the partial
sums of the Walsh-Fourier series with "optimal" weights are bounded from the
martingale Hardy space $H_{1}(G)$ to the space $\text{weak}-L_{1}(G),$ but is
not bounded from $H_{1}(G)$ to the space $L_{1}(G).$

##  2\. Preliminaries

Let $\mathbb{N}_{+}$ denote the set of the positive integers,
$\mathbb{N}:=\mathbb{N}_{+}\cup\\{0\\}.$ Denote by $Z_{2}$ the discrete cyclic
group of order $2$, that is $Z_{2}:=\\{0,1\\},$ where the group operation is
the modulo $2$ addition and every subset is open. The Haar measure on $Z_{2}$
is given so that the measure of a singleton is $1/2$. Define the group $G$ as
the complete direct product of the group $Z_{2},$ with the product of the
discrete topologies of $Z_{2}$‘s. The elements of $G$ are represented by
sequences $x:=(x_{0},x_{1},...,x_{j},...),\ \text{ where }\ x_{k}=0\vee 1.$

It is easy to give a base for the neighborhood of $x\in G:$

$I_{0}\left(x\right):=G,\text{ \ }I_{n}(x):=\\{y\in
G:y_{0}=x_{0},...,y_{n-1}=x_{n-1}\\}\text{ }(n\in\mathbb{N}).$

Denote $I_{n}:=I_{n}\left(0\right),\ \ \overline{I_{n}}:=G\backslash I_{n}.$
Then, it is easy to prove that

(5) $\overline{I_{M}}=\overset{M-1}{\underset{s=0}{\bigcup}}I_{s}\backslash
I_{s+1}.$

Let define the Walsh system $w:=(w_{n}:n\in\mathbb{N})$ on $G$ by

$w_{n}(x):=\overset{\infty}{\underset{k=0}{\Pi}}r_{k}^{n_{k}}\left(x\right)=\left(-1\right)^{\underset{k=0}{\overset{\left|n\right|}{\sum}}n_{k}x_{k}}\text{\qquad}\left(n\in\mathbb{N}\right).$

The Walsh system is orthonormal and complete in $L_{2}(G)$ (see e.g. [26] and
[28]).

If $f\in L_{1}$ we define Fourier coefficients, partial sums of the Fourier
series, Dirichlet kernels with respect to the Walsh system by

$\displaystyle\widehat{f}\left(k\right):=\int_{G}fw_{k}d\mu\
\left(k\in\mathbb{N}\right),\ \
S_{n}f:=\sum_{k=0}^{n-1}\widehat{f}\left(k\right)w_{k},\ \
D_{n}:=\sum_{k=0}^{n-1}w_{k}\ \left(n\in\mathbb{N}_{+}\right).$

Recall that (see [17], [26] and [28])

(6)
$D_{2^{n}}\left(x\right)=\left\\{\begin{array}[]{ll}2^{n},&\,\text{if\thinspace\thinspace\thinspace}x\in
I_{n},\\\ 0,&\text{if}\,\,\ x\notin I_{n},\end{array}\right.$ (7)
$D_{n}=w_{n}\overset{\infty}{\underset{k=0}{\sum}}n_{k}r_{k}D_{2^{k}}=w_{n}\overset{\infty}{\underset{k=0}{\sum}}n_{k}\left(D_{2^{k+1}}-D_{2^{k}}\right),\text{
for \ }n=\overset{\infty}{\underset{i=0}{\sum}}n_{i}2^{i}.$

The $\sigma$-algebra generated by the intervals
$\left\\{I_{n}\left(x\right):x\in G\right\\}$ will be denoted by
$\zeta_{n}\left(n\in\mathbb{N}\right).$ Denote by
$F=\left(F_{n},n\in\mathbb{N}\right)$ martingale with respect to
$\digamma_{n}$ $\left(n\in\mathbb{N}\right)$ (see e.g. [36, 37]). The maximal
function $F^{\ast}$ of a martingale $F$ is defined by
$F^{\ast}:=\sup_{n\in\mathbb{N}}\left|F_{n}\right|.$ For $0<p<\infty$ the
Hardy martingale spaces $H_{p}\left(G\right)$ consists of all martingales for
which

$\left\|F\right\|_{H_{p}}:=\left\|F^{\ast}\right\|_{p}<\infty.$

For every martingale $F=\left(F_{n},n\in\mathbb{N}\right)$ and every
$k\in\mathbb{N}$ the limit

$\widehat{F}\left(k\right):=\lim_{n\rightarrow\infty}\int_{G}F_{n}\left(x\right)w_{k}\left(x\right)d\mu\left(x\right)$

exists and it is called the $k$-th Walsh-Fourier coefficients of $F.$

A bounded measurable function $a$ is called $p$-atom, if there exists a dyadic
interval $I,$ such that

$\int_{I}ad\mu=0,\text{ \ \
}\left\|a\right\|_{\infty}\leq\mu\left(I\right)^{-1/p},\text{ \ \
supp}\left(a\right)\subset I.$

The dyadic Hardy martingale spaces $H_{p}(G)$ for $0<p\leq 1$ have an atomic
characterization. Namely, the following holds (see [26], [36, 37]):

######  Lemma 1.

A martingale $F=\left(F_{n},n\in\mathbb{N}\right)$ belongs to $H_{p}(G)\
\left(0<p\leq 1\right)$ if and only if there exists a sequence
$\left(a_{k},\text{ }k\in\mathbb{N}\right)$ of p-atoms and a sequence
$\left(\mu_{k},k\in\mathbb{N}\right)$ of real numbers such that for every
$n\in\mathbb{N},$

(8) $\qquad\sum_{k=0}^{\infty}\mu_{k}S_{2^{n}}a_{k}=F_{n},\text{ \ \ \
}\sum_{k=0}^{\infty}\left|\mu_{k}\right|^{p}<\infty,$

Moreover,
$\left\|F\right\|_{H_{p}}\backsim\inf\left(\sum_{k=0}^{\infty}\left|\mu_{k}\right|^{p}\right)^{1/p},$
where the infimum is taken over all decomposition of $F$ of the form (8).

##  3\. The main result

Our main result reads:

######  Theorem 1.

a) Let $f\in{{H}_{1}}$. Then the weighted maximal operator
$\widetilde{S}^{\ast,\nabla},$ defined by

(9)
$\widetilde{S}^{\ast,\nabla}:=\underset{k\in\mathbb{N}}{\sup}\frac{\left|S_{n_{k}}F\right|}{V(n_{k})}$

is bounded from the Hardy space ${{H}_{1}}$ to the space $\text{weak}-L_{1}.$

b) Let $f\in{{H}_{1}}$. Then the weighted maximal operator
$\widetilde{S}^{\ast,\nabla},$ defined by (9) is not bounded from the Hardy
space ${{H}_{1}}$ to the space $L_{1}.$

###### Proof.

Since $S_{n}/V(n)$ is bounded from $L_{\infty}$ to $L_{\infty},$ by Lemma 1,
the proof of Theorem 1 will be complete, if we prove that

(10) $t\mu\left\\{x\in\overline{I_{M}}:\widetilde{S}^{\ast,\nabla}a(x)\geq
t\right\\}\leq c_{p}<\infty,\text{ \ \ \ \ }t\geq 0$

for every $p$-atom $a.$ In this parer $c_{p}$ (or $C_{p}$) denotes a positive
constant depending only on $p$ but which can be different in different places.

We may assume that $a$ is an arbitrary $p$-atom, with support $I,\
\mu\left(I\right)=2^{-M}$ and $I=I_{M}.$ It is easy to see that
$S_{n}a\left(x\right)=0,\ \text{ when }\ n<2^{M}.$ Therefore, we can suppose
that $n\geq 2^{M}.$ Since $\left\|a\right\|_{\infty}\leq 2^{M},$ we obtain
that

$\displaystyle\frac{\left|S_{n}a\left(x\right)\right|}{V(n)}$
$\displaystyle\leq$
$\displaystyle\frac{1}{V(n)}\left\|a\right\|_{\infty}\int_{I_{M}}\left|D_{n}\left(x+t\right)\right|\mu\left(t\right)\leq
2^{M}\int_{I_{M}}\left|D_{n}\left(x+t\right)\right|\mu\left(t\right).$

Let $I_{s}\backslash I_{s+1}.$ Then, it is easy to see that $x+t\in
I_{s}\backslash I_{s+1}$ for $t\in I_{M}$ and if we again combine (6) and (7)
we find that $D_{n}\left(x+t\right)\leq c2^{s},\ \text{ for }\ t\in I_{M}$ and

(11) $\displaystyle\frac{\left|S_{n}a\left(x\right)\right|}{V(n)}\leq
c_{p}2^{M}2^{s-M}\leq c_{p}2^{s}.$

By applying (11) for any $x\in I_{s}\backslash I_{s+1},\,0\leq s<M,$ we find
that

(12)
$\displaystyle\widetilde{S}^{\ast,\nabla}a\left(x\right)=\sup_{n\in\mathbb{N}}\left(\frac{\left|S_{n}a\left(x\right)\right|}{2^{\rho\left(n\right)\left(1/p-1\right)}}\right)\leq
C_{p}2^{s}.$

It immediately follows that for $s\leq M$ we have the following estimate

$\widetilde{S}^{\ast,\nabla}a\left(x\right)\leq C_{p}2^{M}\text{ \ \ for any\
\ }x\in I_{s}\backslash I_{s+1},\ \ s=0,1,\cdots,M$

and also that

(13) $\mu\left\\{x\in I_{s}\backslash
I_{s+1}:\widetilde{S}^{\ast,\nabla}a\left(x\right)>C_{p}2^{k}\right\\}=0,\ \ \
k=M,M+1,\ldots$

By combining (5) and (12) we get that

$\displaystyle\left\\{x\in\overline{I_{N}}:\widetilde{S}^{\ast,\nabla}a\left(x\right)\geq
C_{p}2^{k}\right\\}\subset\bigcup_{s=k}^{M-1}\left\\{x\in I_{s}\backslash
I_{s+1}:\widetilde{S}^{\ast,\nabla}a\left(x\right)\geq C_{p}2^{k}\right\\}$

and

(14)
$\displaystyle\mu\left\\{x\in\overline{I_{M}}:\widetilde{S}^{\ast,\nabla}a\left(x\right)\geq
C_{p}2^{k}\right\\}\leq\overset{M-1}{\underset{s=k}{\sum}}\frac{1}{2^{s}}\leq\frac{2}{2^{k}}.$

In view of (13) and (14) we can conclude that

$\displaystyle
2^{k}\mu\left\\{x\in\overline{I_{N}}:\widetilde{S}^{\ast,\nabla}a\left(x\right)\geq{C_{p}}{2^{k}}\right\\}<c_{p}<\infty,$

which shows that (10) holds and the proof of part a) is complete.

Set

$f_{n_{k}}\left(x\right)=D_{2^{n_{k}+1}}\left(x\right)-D_{2^{{n_{k}}}}\left(x\right),\text{
\qquad}n_{k}\geq 3.$

In [5] it was proved that $\left\|f_{n_{k}}\right\|_{H_{p}}\leq 1$ and
$\left|S_{2^{n_{k}}+2^{s}}f_{n_{k}}\left(x\right)\right|\geq c2^{s},$ for
$x\in I_{s+1}\left(e_{s}\right),$ $s=0,\cdots,n_{k}-1.$ Hence,

$\displaystyle\int_{G}\sup_{n\in\mathbb{N}}\frac{\left|S_{n}f_{n_{k}}\right|}{V\left(n\right)}d\mu\geq\overset{n_{k}-1}{\underset{s=0}{\sum}}\int_{I_{s+1}\left(e_{s}\right)}\frac{\left|S_{2^{n_{k}}+2^{s}}f_{n_{k}}\right|}{V\left(2^{n_{k}}+2^{s}\right)}d\mu\geq
c_{p}\overset{n_{k}-1}{\underset{s=0}{\sum}}\frac{1}{2^{s}}2^{s}\geq
C_{p}n_{k}.$

Finally, we get that

$\displaystyle\frac{\left(\int_{G}\left(\sup_{n\in\mathbb{N}}\frac{\left|S_{n}f_{n_{k}}\left(x\right)\right|}{V\left(n\right)}\right)d\mu\left(x\right)\right)^{1/p}}{\left\|f_{n_{k}}\right\|_{H_{p}}}\geq
c_{p}n^{1/p}_{k}\rightarrow\infty,\quad\text{as \quad}k\rightarrow\infty,$

so also part b) is proved and the proof is complete.

∎

##  4\. Final comments and open problem

We finalize this paper with another natural conjecture related to Theorem 1.
For this we need some new characterization of $n\in\mathbb{N}.$

Let
$2^{s}\leq{{n}_{{{s}_{1}}}}\leq{{n}_{{{s}_{2}}}}\leq...\leq{{n}_{{{s}_{r}}}}\leq{2^{s+1}},\
s\in\mathbb{N}.$ For such $n_{s_{j}},$ which can be written as
${{n}_{{{s}_{j}}}}=\sum\limits_{i=1}^{{{r}_{{{s}_{j}}}}}{\sum\limits_{k=l_{i}^{{{s}_{j}}}}^{t_{i}^{{{s}_{j}}}}{2^{k}}},$
where $0\leq l_{1}^{{{s}_{j}}}\leq t_{1}^{{{s}_{j}}}\leq
l_{2}^{{{s}_{j}}}-2<l_{2}^{{{s}_{j}}}\leq t_{2}^{{{s}_{j}}}\leq...\leq
l_{{{r}_{j}}}^{{{s}_{j}}}-2<l_{{{r}_{s_{j}}}}^{{{s}_{j}}}\leq
t_{{{r}_{s_{j}}}}^{{{s}_{j}}},$ we define

(15) $\displaystyle{{A}_{s}}$ $\displaystyle:=$
$\displaystyle{\left\\{l_{1}^{s},l_{2}^{s},...,l_{r^{1}_{s}}^{s}\right\\}}\bigcup{\left\\{t_{1}^{s},t_{2}^{s},...,t_{{{r}_{s}^{2}}}^{s}\right\\}}={\left\\{u_{1}^{s},u_{2}^{s},...,u_{r^{3}_{s}}^{s}\right\\}},$

where $u_{1}^{s}<u_{2}^{s}<...<u_{r^{3}_{s}}^{s}.$ We note that
$t_{{{r}_{s_{j}}}}^{{{s}_{j}}}=s\in{{A}_{s}},\ \ \text{ for }\ \ j=1,2,...,r.$

We denote the cardinality of the set $A_{s}$ by $|A_{s}|$, that is
$card(A_{s}):=|A_{s}|.$ By this definition we can conclude that
$|A_{s}|=r_{s}^{3}\leq r_{s}^{1}+r_{s}^{2}.$ It is evident that
$\sup_{s\in\mathbb{N}}|A_{s}|<\infty$ if and only if the sets
$\\{{{n}_{{{s}_{1}}}},{{n}_{{{s}_{2}}}},...,{{n}_{{{s}_{r}}}}\\}$ are
uniformly finite for all $s\in\mathbb{N}_{+}$ and each ${{n}_{{{s}_{j}}}}$ has
bounded variation $V(n_{s_{j}})<c<\infty,\ \text{for each}\ j=1,2,\ldots,r.$

######  Conjecture 1.

a) Let $f\in{{H}_{1}}$ and $\left\\{n_{k}:k\geq 0\right\\}$ is a sequence of
positive numbers. Then the weighted maximal operator
$\widetilde{S}^{\ast,\nabla},$ defined by

$S^{\ast,\nabla}F:=\underset{k\in\mathbb{N}}{\sup}\frac{\left|S_{n_{k}}F\right|}{A_{|n_{k}|}},$

is bounded from the Hardy space ${{H}_{1}}$ to the Lebesgue space ${{L}_{1}}$.

b) (Sharpness) Let ${{\sup}_{k\in\mathbb{N}}}|{{A}_{n_{k}}}|=\infty$ and
$\\{\varphi_{n}\\}$ is a nondecreasing sequence satisfying the condition
$\overline{\lim}_{k\rightarrow\infty}\left({A_{|n_{k}|}}/{\varphi_{|n_{k}|}}\right)=\infty.$
Then there exists a martingale $f\in H_{1},$ such that the maximal operator,
defined by
$\underset{k\in\mathbb{N}}{\sup}\frac{\left|S_{n_{k}}F\right|}{\varphi_{|n_{k}|}}$
is not bounded from the Hardy space ${{H}_{1}}$ to the Lebesgue space
${{L}_{1}}.$

## References

  * [1] M. Avdispahić and N. Memić, On the Lebesgue test for convergence of Fourier series on unbounded Vilenkin groups, Acta Math. Hungar., 129, 4 (2010), 381-392.
  * [2] N. K. Bary, Trigonometric series. Gos. Izd. Fiz. Mat. Lit. Moscow 1961, (Russian).
  * [3] D. Baramidze, Martingale Hardy spaces and some new weighted maximal operators of partial sums of Walsh-Fourier series, Georgian Math. J., (to appear).
  * [4] D. Baramidze, L.-E. Persson and G. Tephnadze, Some new $(H_{p}-L_{p})$ type inequalities for weighted maximal operators of partial sums of Walsh-Fourier series, Positivity, 2023, 27(3), 38.
  * [5] D. Baramidze, L.-E. Persson, H. Singh and G. Tephnadze, Some new weak $(H_{p}-L_{p})$ type inequality for weighted maximal operators of partial sums of Walsh-Fourier series, Mediterr. J. Math., 20, 5, (2023) 284.
  * [6] D. Baramidze, L.-E. Persson, H. Singh and G. Tephnadze, Some new results and inequalities for subsequences of Nörlund logarithmic means of Walsh-Fourier series, J. Inequal. Appl., (2022), paper no. 30, 13 pp.
  * [7] I. Blahota, Approximation by Vilenkin-Fourier sums in $L_{p}(G_{m})$, Acta Acad. Paed. Nyireg., 13 (1992), 35-39.
  * [8] I. Blahota and K. Nagy, Approximation by $\theta$-means of Walsh-Fourier series, Anal. Math., 44, (2018), 57-71.
  * [9] I. Blahota, K. Nagy and G. Tephnadze, Approximation by Marcinkiewicz $\theta$-means of double Walsh-Fourier series, Math. Inequal. Appl., 22, 3 (2019) 837-853.
  * [10] I. Blahota, K. Nagy, L. E. Persson and G. Tephnadze, A sharp boundedness result concerning maximal operators of Vilenkin-Fourier series on martingale Hardy spaces, Georgian Math. J., 26, 3 (2019), 351-360.
  * [11] I. Blahota, L. E. Persson and G. Tephnadze, Two-sided estimates of the Lebesgue constants with respect to Vilenkin systems and applications, Glasg. Math. J., 60, 1 (2018), 17-34.
  * [12] N. I. Fine, On Walsh function, Trans. Amer. Math. Soc. 65 (1949), 372-414.
  * [13] S. Fridli, Approximation by Vilenkin-Fourier series, Acta Math. Hung., 47, 1-2 (1986), 33-44.
  * [14] G. Gát, Best approximation by Vilenkin-like systems, Acta Acad. Paed. Nyireg., 17 (2001), 161-169.
  * [15] G. Gát, U. Goginava and G. Tkebuchava, Convergence in measure of logarithmic means of quadratical partial sums of double Walsh-Fourier series, J. Math. Appl., 323, 1 (2006), 535-549.
  * [16] U. Goginava and G. Tkebuchava, Convergence of subsequence of partial sums and logarithmic means of Walsh-Fourier series, Acta Sci. Math., 72 (2006), 159-177.
  * [17] B. I. Golubov, A. V. Efimov and V. A. Skvortsov, Walsh series and transforms. (Russian) Nauka, Moscow, 1987, English transl, Mathematics and its Applications (Soviet Series), 64. Kluwer Academic Publishers Group, Dordrecht, 1991.
  * [18] N. V. Gulićev, Approximation to continuous functions by Walsh-Fourier series, Anal. Math., 6 (1980), 269-280.
  * [19] S. F. Lukomskii, Lebesgue constants for characters of the compact zero-dimensional Abelian groups, East J. Approx., 15, 2 (2009), 219-231.
  * [20] N. Memić, I. Simon and G. Tephnadze, Strong convergence of two-dimensional Vilenkin-Fourier series, Math. Nachr., 289, 4 (2016) 485-500.
  * [21] K. Nagy, Approximation by Cesáro means of negative order of Walsh-Kaczmarz-Fourier series, East J. Approx., 16, 3 (2010), 297-311.
  * [22] K. Nagy and G. Tephnadze, The Walsh-Kaczmarz-Marcinkiewicz means and Hardy spaces, Acta Math. Hung., 149, 2 (2016), 346-374.
  * [23] C.W. Onneweer, On $L$-convergence of Walsh-Fourier series, Internat. J. Math. Sci., 1 (1978), 47-56.
  * [24] A. Paley, A remarkable series of orthogonal functions, Proc. London Math. Soc. 34 (1932), 241-279.
  * [25] L. E. Persson, G. Tephnadze and P. Wall, On an approximation of 2-dimensional Walsh-Fourier series in the martingale Hardy spaces, Ann. Funct. Anal., 9, 1 (2018), 137-150.
  * [26] L. E. Persson, G. Tephnadze and F. Weisz, Martingale Hardy Spaces and Summability of One-dimenional Vilenkin-Fourier Series, Birkhäuser/Springer, 2022.
  * [27] L.-E. Persson, F. Schipp, G. Tephnadze and F. Weisz, An analogy of the Carleson-Hunt theorem with respect to Vilenkin systems, J. Fourier Anal. Appl., 28, 48 (2022), 1-29.
  * [28] F. Schipp, W. Wade, P. Simon and J. Pál, Walsh series, An Introduction to Dyadic Harmonic Analysis, Akademiai Kiado, Budapest-Adam-Hilger, Bristol-New-York, 1990.
  * [29] P. Simon, A note on the of the Sunouchi operator with respect to Vilenkin systems, Ann. Univ. Sci. Budapest. Eotvos. Sect. Math., 43 (2001), 101-116.
  * [30] G. Tephnadze, On the Vilenkin-Fourier coefficients, Georgian Math. J., 20, 1 (2013), 169-177.
  * [31] G. Tephnadze,  On the partial sums of Vilenkin-Fourier series, J. Contemp Math. Anal., 49, 1 (2014), 23-32.
  * [32] G. Tephnadze,  On the partial sums of Walsh-Fourier series, Colloq. Math., 141, 2 (2015), 227-242.
  * [33] G. Tephnadze,  On the convergence of partial sums with respect to Vilenkin system on the martingale Hardy spaces, J. Contemp. Math. Anal., 53, 5 (2018) 294-306.
  * [34] G. Tephnadze,  Convergence and Strong Summability of the two-dimensional Vilenkin-Fourier Series, Nonlinear Studies, 26, 4, (2019) 973-989.
  * [35] G. Tephnadze,  Strong convergence of two-dimensional Walsh-Fourier series, Ukr. Math. J., 65, 6 (2013), 914-927.
  * [36] F. Weisz, Martingale Hardy spaces and their applications in Fourier Analysis, Springer, Berlin-Heideiberg-New York, 1994.
  * [37] F. Weisz, Hardy spaces and Cesáro means of two-dimensional Fourier series, Bolyai Soc. Math. Studies, 5 (1996), 353-367.