# Revamping Multilingual Agreement Bidirectionally via Switched Back-
translation for Multilingual Neural Machine Translation

Hongyuan Lu♡, Haoyang Huang♠, Dongdong Zhang♠,
Furu Wei♠, Wai Lam♡
♡The Chinese University of Hong Kong
♠Microsoft Corporation
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Despite the fact that multilingual agreement (MA) has shown its importance for
multilingual neural machine translation (MNMT), current methodologies in the
field have two shortages: (i) require parallel data between multiple language
pairs, which is not always realistic and (ii) optimize the agreement in an
ambiguous direction, which hampers the translation performance. We present
Bidirectional Multilingual Agreement via Switched Back-translation (BMA-SBT),
a novel and universal multilingual agreement framework for fine-tuning pre-
trained MNMT models, which (i) exempts the need for aforementioned parallel
data by using a novel method called switched BT that creates synthetic text
written in another source language using the translation target and (ii)
optimizes the agreement bidirectionally with the Kullback-Leibler Divergence
loss. Experiments indicate that BMA-SBT clearly improves the strong baselines
on the task of MNMT with three benchmarks: TED Talks, News, and Europarl. In-
depth analyzes indicate that BMA-SBT brings additive improvements to the
conventional BT method.111Code and data will be available upon publication.

##  1 Introduction

Conventional multilingual neural machine translation (MNMT) leverages
independent parallel data during the training process. In comparison, the
multilingual agreement (MA) explicitly minimizes the output difference between
two source inputs written in different languages but with the same meaning.
Despite its success in from-scratch training on MT (Yang et al., 2021c),
current methodologies suffer from at least two disadvantages that limit their
scope of usage. Firstly, conventional MA leverages word alignment tools to
create code-switching sentence-level data (Yang et al., 2021c). This process
usually requires authentic parallel data between multiple language pairs. For
example, assuming we would like to enhance Chinese to English and German to
English, conventional MA assumes the existence of parallel data from Chinese
to German, which however sometimes does not exist. Secondly, the direction of
agreement-based learning can be bidirectional (Zhang et al., 2019), while the
direction of conventional multilingual agreement is usually ambiguous.
However, since languages usually have different linguistic clues and they are
helpful to each other, we argue that optimizing the multilingual agreement
explicitly in a bidirectional manner can help the languages to learn from each
other and hence further enhance cross-lingual learning.

Figure 1: An illustrated example that can be benefited from Multilingual
Agreement optimized in a bi-direction manner. The words in green are the
correct translation, and the words in red are the wrong translation. Here,
Chinese is incorrectly translated since it does not have past tense for verbs,
and German is incorrectly translated due to the shared subword unit with
different meanings between Glanz (German, shine) and Glance (English, take a
brief look at). Best viewed in colour.

Figure 1 depicts such a case that can be benefited from bidirectionally
enhanced MA. The underlying reason is that both of the source inputs have
cross-lingual ambiguities here. Since Chinese does not have past tense verbs,
it is intuitive to use some auxiliary languages with past tense. Furthermore,
since German shares partial vocabulary subwords with English under MNMT, this
introduces cross-lingual ambiguities and using a language that does not share
its vocabulary subwords with English, e.g., such as Chinese, could be helpful.

As a side note, since MA was proposed as a method for from-scratch training
for MT, it was unclear whether conventional MA is also effective as a fine-
tuning technique for pre-trained models.

Furthermore, how to appropriately apply back-translation to a multilingual
setting is also an under-studied subject despite its importance.

This paper proposes BMA-SBT, a novel MNMT framework that (i) exempts the need
for parallel data between multiple language pairs and (ii) optimizes the MA in
a bidirectional manner. To exempt the need for parallel data, we propose
switched back-translation to produce synthetic text in some different
auxiliary source languages with the translation target.222For example, Chinese
as the source, English as the target, and Japanese as the auxiliary source. To
optimize the MA in an explicit bidirectional manner, we use a bidirectional
Kullback–Leibler Divergence loss instead of the code-switching for
conventional MA. This enforces the original source language and the synthetic
auxiliary language to have the same outputs as the target reference
translation in a bidirectional manner.

We conduct experiments on three MT benchmarks: TED Talks (Cettolo et al.,
2015), News benchmark (News-commentary) and Europarl (Koehn, 2005).
Experimental results indicate that BMA-SBT clearly improves the strong pre-
trained baselines on all three benchmarks. In-depth analyses indicate that
BMA-SBT effectively mitigates cross-lingual ambiguities.

In summary, we make three key contributions:

  * •

This paper proposes a novel framework called BMA-SBT, the first MNMT framework
that achieves MA without the requirement of extra parallel data and explicitly
optimizes the MA in a bidirectional manner.

  * •

BMA-SBT yields clear improvement on SOTA pre-trained MT model on three MT
benchmarks: TED Talks, News, and Europarl.

  * •

We conduct in-depth analyses of BMA-SBT. Results indicate that BMA-SBT brings
additive improvement to conventional BT and bidirectionality is important for
MA.

Also, this is the first work that demonstrates the usefulness of MA as a fine-
tuning technique.

Figure 2: Overview of our proposed BMA-SBT framework. $x$ and $y$ denote the
original source and target text written in the source language $L_{s}$ and
target language $L_{t}$. $\tilde{z}$ denotes the synthetic text translated
from the original target text into language $L_{a}$. $\tilde{y}$ denotes the
translation output from the original source text produced by the multilingual
Transformer and $\hat{y}$ denotes the translation output from the synthetic
text. The letters with subscripts such as $x_{i}$ denote the $i$-th token in
the original source text. The red arrows denote the backward gradient flow
computed by the bidirectional KL loss that updates the shared multilingual
Transformer encoder and decoder. Best viewed in colour.

##  2 Bidirectional Multilingual Agreement via Switched Back-translation

###  2.1 Multilingual Neural Machine Translation

We conduct our experiments on the task of MNMT on large-scale pre-trained
multilingual translation model (Yang et al., 2021a; Lu et al., 2022) that
handles multiple languages by sharing a universal subword dictionary among all
the languages. For both training and inference, given $I$ languages
$\\{L_{1},,...,L_{I}\\}$, we prefix a special target language token $L_{t}$ to
the source inputs to signal the multilingual model that we are translating
from an arbitrary source language to the target language $L_{t}$.

Given a bilingual dataset for machine translation that consists of
$\mathcal{N}$ training instances
$\\{\mathcal{T}_{1},,...,\mathcal{T}_{\mathcal{N}}\\}$, each of the bilingual
translation pairs $\mathcal{T}_{i}$ in the source bilingual dataset
$\mathcal{D}_{\mathcal{M}}$ contains a source input $x$ and the corresponding
translation target $y$. With a Seq2Seq generation model (Sutskever et al.,
2014) with parameters $\theta$, we train the model by optimizing the following
likelihood:

$\mathcal{L}_{main}=\sum_{n=1}^{\mathcal{N}}\mathbb{E}_{x_{n},y_{n}\in\mathcal{D}_{\mathcal{M}}}[-\log
P_{\theta}(y\mid x)],$ (1)

where $\mathcal{L}_{main}$ denotes the standard training loss that we adopt
for MNMT.

###  2.2 BMA-SBT

In this subsection, we introduce our novel framework Bidirectional
Multilingual Agreement via Back-translation (BMA-SBT). Compared to the
conventional multilingual agreement, BMA-SBT exempts the need for parallel
data and specifies the direction of the multilingual agreement in a
bidirectional manner. We first introduce how we use BT to create synthetic
parallel data which are appropriate for the use of the multilingual agreement,
and we then introduce how to leverage KL divergence loss to make the
multilingual agreement bidirectional.

#### Switched Back-translation

The conventional multilingual agreement (MA) requires authentic parallel data,
which could be commonly unrealistic in a real-world setting. Formally, for the
translation pair $x$ and $y$ in Equation 1, conventional MA requires another
instance $z$, which is written in a different language and in the equivalent
meaning to $x$ and $y$. And this process was designed and experimented on
from-scratch training. These facts limit the use of the conventional MA.

To mitigate the above-mentioned shortages, we propose a novel method called
switched back-translation that creates synthetic text $\tilde{z}$ written in
different source languages by feeding the translation target $y$ to a machine
translation model through back-translation.333While we can use the translation
source $x$ to create $\tilde{z}$, we empirically have found that this degrades
the improvement. We postulate that if the source text has ambiguities, then
this is less helpful to create the auxiliary text with the source text. Note
that $\tilde{z}$, $x$, and $y$ are equivalent in their meanings, but they are
written in different languages.

This helps us to establish a synthetic bilingual auxiliary dataset
$\mathcal{D}_{\mathcal{A}}$ that is consisted of $\mathcal{M}$ training
instances. We then train the multilingual model by maximising the following
likelihood:

$\mathcal{L}_{auxiliary}=\sum_{n=1}^{\mathcal{M}}\mathbb{E}_{\tilde{z}_{n},y_{n}\in\mathcal{D}_{\mathcal{A}}}[-\log
P_{\theta}(y\mid\tilde{z})].$ (2)

We also differentiate the switched back-translation we propose here from the
conventional BT. For BT which was originally proposed for bilingual MT
(Sennrich et al., 2016), we usually obtain $x^{\prime}$ from the original
monolingual target $y$, where $x^{\prime}$ should be written in the same
source language in our interest. In contrast, BMA-SBT creates $\tilde{z}$ that
should have the equivalent meaning as $y$, but it should be written in
different languages from both the original source and target languages for the
purpose of applying the multilingual agreement.444For a fair comparison, we
use the Baseline Model and the monolingual English sentences in the downstream
dataset for data augmentation with BT (Sennrich et al., 2016) and SBT.

In conclusion, this evolves the conventional MA into a universal fine-tuning
technique for MNMT which does not need extra parallel data. BMA-SBT fits the
real-world setting and can be applied with some modifications to other
generation tasks for cross-lingual learning.

#### Bidirectional Multilingual Agreement

The direction for agreement-based learning can be bidirectional (Zhang et al.,
2019). However, the conventional multilingual agreement has an ambiguous
direction due to the nature of code-switching. By using parallel data,
conventional MA constructs code-switching data $c$ from $x$ and $z$, which
denotes the translation source and the authentic auxiliary text respectively.
Note that $x$ and $z$ have the same meaning to the translation target $y$, but
they are written in different languages. The code-switching is then done with
a word alignment tool between $x$ and $z$ at the word level, usually with a
low code-switching replacement ratio as low as 10% (Yang et al., 2021c).
Formally, conventional MA trains MNMT by maximising the following likelihood:

$\mathcal{L}_{MA}=\sum_{n=1}^{\mathcal{Q}}\mathbb{E}_{c,y_{n}\in\mathcal{D}_{\mathcal{C}}}[-\log
P_{\theta}(y\mid c)],$ (3)

where $y$ denotes the translation target, $\mathcal{D}_{\mathcal{C}}$ denotes
the code-switching dataset automatically constructed, and $\mathcal{Q}$
denotes the number of samples in the code-switching dataset.

In addition to the fact that conventional MA requires authentic data $z$ which
is not always realistic, we also argue that code-switching optimizes in an
ambiguous direction, usually with a low code-switching ratio as low as 10%.
Therefore, we consider that cross-lingual learning could be less efficient in
this manner. As depicted in Figure 1, MNMT can be benefited by encouraging
multilingual agreement in a bidirectional manner. Hence, we use a KL
divergence loss to specify the direction of multilingual agreement in a clear
bidirectional manner. Since the authentic parallel text $z$ is not always
available, we use the aforementioned synthetic auxiliary text $\tilde{z}$ to
calculate a bidirectional MA (BMA) divergence loss:

$\mathcal{L}_{BMA}=\alpha\mathcal{L}_{KL_{1}}+(1-\alpha)\mathcal{L}_{KL_{2}},$
(4)

Model | Fr$\rightarrow$En | De$\rightarrow$En | Zh$\rightarrow$En | Vi$\rightarrow$En | Cs$\rightarrow$En | Th$\rightarrow$En | Avg.
---|---|---|---|---|---|---|---
Sentence-level Systems
HAN${\dagger}$ | - | - | 24.00 | - | - | - | -
M2M-100 | 50.18 | 42.24 | 26.62 | 34.92 | 37.84 | 27.28 | 36.51
mBART | 48.69 | 44.80 | 28.39 | 37.18 | 39.47 | - | -
Baseline Model + BT | 50.69 | 47.07 | 30.35 | 39.59 | 43.05 | 32.30 | 40.51
Document-level Systems
mT5${\dagger}$ | - | - | 24.24 | - | - | - | -
M2M-100 | 49.43 | 43.82 | 26.63 | 35.91 | 39.04 | 25.93 | 36.79
mBART | 49.16 | 44.86 | 29.60 | 37.09 | 39.64 | - | -
MARGE${\dagger}$ | - | - | 28.40 | - | - | - | -
Baseline Model + BT | 49.53 | 45.98 | 30.17 | 39.28 | 42.33 | 30.62 | 39.65
Multilingual Agreement | 48.99 | 47.34 | 30.35 | 39.79 | 43.01 | 32.14 | 40.27
Systems with Bilingual Parallel Document Data for Pre-training
DOCmT5${\dagger}$ | - | - | 31.40∗ | - | - | - | -
BMA-SBT (Ours) + BT | 51.10 | 47.59 | 30.80 | 40.20 | 43.17 | 32.23 | 40.85
Ablation Study
\- w/o $KL_{1}$ | 49.58 | 46.38 | 29.46 | 39.09 | 42.87 | 30.59 | 39.66
\- w/o $KL_{2}$ | 50.56 | 47.47 | 30.26 | 40.02 | 43.15 | 31.89 | 40.56
\- w/o $KL_{1}$&$KL_{2}$ | 49.73 | 46.64 | 30.58 | 39.81 | 42.85 | 32.06 | 40.28
Table 1:  Test results on TED Talks in the direction of (X $\rightarrow$ En).
${\dagger}$: scores are taken from the official papers for these models. -:
the scores are not reported or the language is not supported. *: the score is
not directly comparable due to the use of document-level parallel corpora for
pre-training. The Baseline Model refers to the model described in Section 3.1,
which is used for parameter initialization for BMA-SBT. BT refers to the
conventional back-translation method described in Section 2.2. $KL_{1}$ and
$KL_{2}$ refers to the loss described in Equation 5 and Equation 6
respectively. We train our system BMA-SBT at the document level.

where $\mathcal{L}_{KL_{1}}$ and $\mathcal{L}_{KL_{2}}$ represents the KL
divergence loss in two directions:

$\mathbb{E}[KL(P_{\theta}(y\mid x)\mid\mid P_{\theta}(y\mid\tilde{z}))]$ (5)

for $\mathcal{L}_{KL_{1}}$, which means that we enforce the original source
text $x$ to learn from the synthetic $\tilde{z}$. Note that $x$ and
$\tilde{z}$ have the same meaning, but they are written in different
languages. We also optimize in the other direction:

$\mathbb{E}[KL(P_{\theta}(y\mid\tilde{z})\mid\mid P_{\theta}(y\mid x))]$ (6)

for $\mathcal{L}_{KL_{2}}$.555Empirically, we have found that setting a
balanced value with $\alpha=0.5$ brings a good performance. In contrast to
$KL_{1}$, this means that the synthetic text $\tilde{z}$ should learn from the
original text $x$. Bidirectionality is necessary to enforce both languages to
learn from each other. Here, $x$ and $y$ denote the original translation
source and target respectively, and $\tilde{z}$ denotes the synthetic
auxiliary text created by BMA-SBT via translation.

#### BMA-SBT

Overall, we propose a novel BMA-SBT framework that optimizes the MNMT models
with the following combinatory loss:

$\mathcal{L}_{BMA-
SBT}=\mathcal{L}_{main}+\mathcal{L}_{auxiliary}+\mathcal{L}_{BMA}$ (7)

Figure 2 depicts the overview of BMA-SBT. The final KL loss at the right edge
of the figure refers to $\mathcal{L}_{BMA}$, $\mathcal{L}_{main}$ is
calculated with the training instance at the top, and
$\mathcal{L}_{auxiliary}$ is calculated with the training instance at the
bottom.

BMA-SBT can be improved with multiple auxiliary languages for agreement in an
ensemble manner. This requires more tuning and computational costs. We leave
this to future work.

##  3 Experiments

###  3.1 Implementation Details

#### Model Configuration

The Transformer architecture we use is composed of 24 encoder layers and 12
interleaved decoder layers. Furthermore, the architecture has an embedding
size of 1024, with a dropout rate of 0.1. The feed-forward network has a size
of 4096, with 16 attention heads. For parameter initialization, we follow Ma
et al. (2021) and Yang et al. (2021b) to pre-train a strong MT system with
sentence-level bilingual data. For the rest of this paper, We call it the
Baseline Model and use it as a strong baseline system.

#### Data Pre-processing

For all of the experiments conducted in this paper, we use SentencePiece (Kudo
and Richardson, 2018) for tokenization. The SentencePiece model we use is the
same as Yang et al. (2021b). Also, we follow prior works to prefix the source
input translation texts with a language tag that indicates the target language
of the outputs.

#### Evaluations

We use the BLEU scores (Papineni et al., 2002) computed with the script from
SacreBLEU for evaluation.666https://github.com/mjpost/sacrebleu

#### Training Details

We use the Adam optimizer (Kingma and Ba, 2014) and set it with the
hyperparameter $\beta_{1}=0.9$ and $\beta_{2}=0.98$ for downstream fine-
tuning. We set the learning rate as $1e\text{-}5$, with a warmup step of
$4000$. We use the label smoothing cross-entropy for the standard translation
loss and we set label smoothing with a ratio of $0.1$ for model training. All
of the fine-tuning experiments reported in this paper are conducted on 8
NVIDIA V100 GPUs. We set the batch size as 512 tokens per GPU. Furthermore, to
simulate a larger batch size, we update the models every 128 steps. For
bilingual back-translation models, we use the downstream datasets for training
on the same Transformer architecture.

###  3.2 TED Talks

#### Experimental Settings

We use the IWSLT15 Campaign for the evaluation of TED Talks, on the task of
multilingual MT. Prior systems have reported scores on only 1 or 2 translation
directions (Lee et al., 2022; Sun et al., 2022), and Lee et al. (2022)
supports only the translation direction into English (X $\rightarrow$ En). We
report a wider range of language directions on the benchmark. We split all
documents into sub-documents with a maximum of 512 tokens for all
train/dev/test sets during training and inference. We use the official
parallel training data from IWSLT15 with no additional monolingual data and
the official 2010 dev set and 2010-2013 test set for evaluation (Liu et al.,
2020; Lee et al., 2022). We use the Baseline Model to generate all the BT data
and the SBT data used for multilingual agreement in BMA-SBT. We fine-tune our
model BMA-SBT at the document level. We report d-BLEU (Liu et al., 2020) using
SacreBLEU.777https://github.com/mjpost/sacrebleu d-BLEU score is a BLEU score
for documents.

Model |  Fr$\rightarrow$En |  De$\rightarrow$En |  Zh$\rightarrow$En |  Cs $\rightarrow$En |  Avg.
---|---|---|---|---|---
Sentence-level Systems
M2M-100 (Fan et al., 2022) | 31.58 | 25.65 | 18.47 | 28.17 | 25.97
mBART (Liu et al., 2020) | 29.93 | 29.31 | 18.33 | 30.15 | 26.93
Document-level Systems
M2M-100 (Fan et al., 2022) | 32.67 | 25.78 | 17.85 | 29.06 | 26.34
mBART (Liu et al., 2020) | 30.14 | 26.35 | 15.01 | 29.79 | 25.32
Baseline Model (Yang et al., 2021b) \+ BT | 36.38 | 34.24 | 25.58 | 36.97 | 33.29
BMA-SBT (Ours) + BT | 37.26 | 34.58 | 26.31 | 37.58 | 33.93
Table 2:  Test results on the News benchmark in the direction of (X
$\rightarrow$ En).

#### Baseline Systems

We report strong baselines evaluated at both sentence and document levels.
Evaluating at the sentence level means that we split documents into sentences
for training and inference. In contrast, evaluating at the document level
means that we split all documents into sub-documents with a maximum of 512
tokens as described in the Experimental Settings. We compare to the following
baselines: M2M-100 (Fan et al., 2022), mBART (Liu et al., 2020),
HAN${\dagger}$ (Yang et al., 2016), MARGE${\dagger}$ (Lewis et al., 2020), and
the Baseline Model that we use to initialize the weights for BMA-SBT.
${\dagger}$: the scores are taken from existing papers. We also report
performance with Multilingual Agreement (Yang et al., 2021c) fine-tuned on
Baseline Model with BT using synthetic parallel text. For a fair comparison,
we do not directly compare to the SOTA model DOCmT5${\dagger}$ (Lee et al.,
2022), as it uses a large amount of bilingual parallel document data for a
document-level multilingual pre-training. The corpus used by DOCmT5 is not
publicly available yet, and our methodology does not make use of such data.
See Appendix A for the number of model parameters.

#### Results

Model | Da$\rightarrow$En | De$\rightarrow$En | El$\rightarrow$En | Es$\rightarrow$En | Fr$\rightarrow$En | It$\rightarrow$En | Nl$\rightarrow$En | Pt$\rightarrow$En | Sv$\rightarrow$En
---|---|---|---|---|---|---|---|---|---
Sentence-level Systems
M2M-100 | 50.40 | 47.38 | 52.28 | 52.03 | 48.26 | 49.70 | 46.78 | 49.84 | 52.34
Baseline Model + BT | 48.94 | 47.25 | 53.46 | 50.57 | 47.68 | 49.49 | 45.95 | 50.65 | 52.77
Document-level Systems
M2M-100 | 50.33 | 47.00 | 52.24 | 52.14 | 48.13 | 49.71 | 46.65 | 40.68 | 52.28
Baseline Model + BT | 49.85 | 47.64 | 53.34 | 51.32 | 48.46 | 50.26 | 47.12 | 50.13 | 52.42
BMA-SBT (Ours) + BT | 50.52 | 47.86 | 54.06 | 52.17 | 48.77 | 50.67 | 47.90 | 50.69 | 52.96
Table 3:  Test results on the Europarl benchmark in the direction of (X
$\rightarrow$ En).

Table 1 presents the evaluation results of TED Talks in the direction of (X
$\rightarrow$ En). BMA-SBT clearly surpasses the baselines. BMA-SBT surpasses
the Baseline Model when both are fine-tuned at the document level by an
average of 1.20 points in the score. BMA-SBT surpasses the Baseline Model
fine-tuned at the sentence level by an average of 0.34 points in the score.
Here, the Baseline Model fine-tuned at the document level is no better than
that of the sentence level. We postulate that the underlying reason is that
previous works have reported that directly optimizing the MNMT model at the
document level can be challenging due to the long input problem (Koehn and
Knowles, 2017). For a fair comparison, we add the conventional back-
translation (BT) to both BMA-SBT and the Baseline Model. See Section 2.2 for
more explanation on the difference between BT and the SBT methods used to
achieve multilingual agreement.

In addition to the fact that BMA-SBT clearly improves the Baseline Model,
which is a strong pre-trained MT system, BMA-SBT also beats other baselines
such as HAN, M2M-100, mT5, and mBART, both fine-tuned at the sentence level
and at the document level. Indeed, the Baseline Model itself is already quite
competitive with these models, and being able to improve such a model is a
piece of clear evidence for the effectiveness of BMA-SBT. The final results we
obtain are close to the SOTA system DOCmT5, which uses a large amount of
bilingual document translation pairs for multilingual pre-training.

#### Ablation Study

The ablation study in Table 1 supports three points of view: (i) the
bidirectionality of the multilingual agreement is necessary, (ii) the
synthetic additional parallel data created by the BT used for MA is useful,
and (iii) BMA-SBT brings additional improvements to the BT.

Source |  ……，当光在西红柿上走过时，它一直在闪耀。它并没有变暗。为什么？因为西红柿熟了，并且光在西红柿内部反射， ……
---|---
Reference |  …, as the light washes over the tomato, It continues to glow. It doesn’t become dark. Why is that? Because the tomato is actually ripe, and the light is bouncing around inside the tomato, …
Google Translate |  …, as the light passed over the tomatoes, It kept shining. It didn’t get darker. Why? Because the tomatoes are ripe, and light is reflected inside the tomatoes, …
Microsoft Translator |  …, as the light walks over the tomatoes, It keeps shining. It didn’t darken. Why? Because the tomatoes are ripe, and light is reflected inside the tomatoes, …
DeepL Translate |  …, as the light traveled over the tomatoes, it kept shining. It doesn’t dim. Why? Because the tomatoes are ripe and the light is reflecting inside the tomatoes, …
Baseline Model (Sentence-level) |  …, as the light goes over the tomato, It’s always glowing. It’s not darkening. Why? Because the tomato is ripe, and light is reflected inside the tomato, …
Baseline Model (Document-level) |  …, as the light passes over the tomato, It keeps flashing. It doesn’t get darker. Why? Because the tomatoes are ripe , and the light is is reflected inside the tomato, …
BMA-SBT |  …, as the light passes over the tomato, It’s always shining. It’s not darkening. Why? Because the tomato is ripe, and the light is reflected inside the tomato, …
Table 4:  A Chinese-to-English case study from TED Talks demonstrates that
BMA-SBT captures better noun-related issues. We highlight the correct
translation in cyan (darker one when printed in B&W), and the mistakes in lime
(lighter one when printed in B&W). Google Translate:
https://translate.google.com/, Microsoft Translator:
https://www.bing.com/translator, DeepL Translate:
https://www.deepl.com/translator.

Firstly, the row of (-w/o $KL_{1}$) and the row of (-w/o $KL_{2}$) represent
the ablations when the KL loss in the directions described in Equation 5 and
Equation 6 are ablated respectively. Here, we can see that both lead to a
degradation in the results. Clearly, using $KL_{2}$ solely without $KL_{1}$
seems to degrade the performance. This is not surprising, as $KL_{1}$ pushes
the output distributions of authentic data to be close to that of auxiliary
text, which helps the model to use more linguistic clues in the auxiliary
text. Also, using $KL_{2}$ solely pushes the outputs of synthetic auxiliary
data to be close to that of the authentic data unidirectionally, which can be
less helpful to the original authentic data. Removing $KL_{2}$ and using
$KL_{1}$ solely also degrades the results, which aligns with our original
motivation depicted for the bidirectionality as in Figure 1.

Secondly, the row of (- w/o $KL_{1}$&$KL_{2}$) brings improvements compared to
Baseline Model + BT, which means that the auxiliary parallel data itself
created by switched back-translation is useful.

Finally, BMA-SBT + BT brings clear improvements to the Baseline Model + BT.
Since both models have used the conventional BT (See Section 3.1 for more
details), the comparison is fair, which means that the BMA-SBT framework is
effective and brings additive improvement to BT.

###  3.3 News

#### Experimental Settings

For evaluation on the News benchmark, we use News Commentary v11 as the
training set, following Sun et al. (2022). We employ newstest2015 as the dev
set, and newstest2016/newstest2019 as the test set respectively for Cs and De.
We use newstest2013 as the dev set and newstest2015 as the test set for Fr. We
use newstest2019 as the dev set and newstest2020 as the test set for Zh. The
remaining settings follow the same as the evaluation on TED Talks.

#### Baseline Systems

As the weights for DOCmT5 are not available at the time of writing, we compare
our system to various strong baselines such as M2M-100, mBART and the Baseline
Model. We run the fine-tuning process on the official checkpoints to obtain
the scores. For a fair comparison, we apply BT to the Baseline Model.

#### Results

Table 2 compares BMA-SBT to strong baselines, and we see that the improvements
with BMA-SBT are clear, and the final results surpass all the strong
baselines. This validates BMA-SBT’s effectiveness as a novel framework.

###  3.4 Europarl

#### Experimental Settings

For the Europarl dataset (Koehn, 2005), we use Europarl-v7 Sun et al. (2022).
W experiment with (X $\rightarrow$ En) where we test nine languages: Da, De,
El, Es, Fr, It, Nl, Pt, and Sv. Like previous works (Bao et al., 2021; Sun et
al., 2022), the dataset is randomly partitioned into train/dev/test divisions,
and we split by English document IDs to avoid information leakage to better
support the multilingual setting.

#### Baseline Systems

As the weights for DOCmT5 are not available at the time of writing, we compare
our system to various strong baselines such as M2M-100 and the Baseline Model.
We run the fine-tuning process on the official checkpoints to obtain the
scores. For a fair comparison, we apply BT to the Baseline Model.

#### Results

Table 3 compares BMA-SBT to strong baselines, and we see that the improvements
with BMA-SBT are obvious, and the final results surpass all the strong
baselines.

###  3.5 Case Study

Table 4 depicts a Zh$\rightarrow$En case study on TED Talks. In addition to
the Baseline Models, we also compare BMA-SBT to various commercial systems
such as Google Translate. In this case, we see that the Chinese text does not
differentiate plural from single. Among all cases, it is clear that BMA-SBT
works the best and can effectively resolve such ambiguity. We also observe
that BMA-SBT perfectly capture the context and attaches the definite article
‘the’ to ‘light’. This aligns with our original intention depicted in Figure 1
to help the models to improve cross-lingual learning via BMA-SBT.

###  3.6 Coherence and Consistency Evaluation

Figure 3 depicts the evaluations in the averaged scores from six translation
directions on TED Talks with BlonDe scores (Jiang et al., 2022). BlonDe is an
evaluation metric designed for MT which considers document-level coherence and
consistency issues that require the model to resolve cross-lingual
ambiguities. We see that BMA-SBT brings effective improvements to the metric.

##  4 Related Work

###  4.1 Multilingual Neural Machine Translation

Conventional bilingual machine translation models deal with two languages: one
as the input, and one as the output. In comparison, multilingual neural
machine translation (MNMT) has achieved great success in handling multiple
languages with a single model. Recently, there have been many pre-training
works on MNMT through multilingual pre-training models that leverage
unsupervised pre-training objectives on monolingual corpora in many different
languages (Conneau et al., 2020; Liu et al., 2020; Xue et al., 2021).
Following the calls that the unsupervised scenario is not strictly realistic
for cross-lingual learning (Artetxe et al., 2020), subsequent works use
parallel corpora with translation pairs for multilingual pre-training (Reid
and Artetxe, 2022; Lee et al., 2022).

While pre-training has shown great success for MNMT (NLLB-Team, 2022), it is
unclear whether the previous methods for from-scratching training on MNMT are
still useful on pre-trained models. Multilingual agreement (Yang et al.,
2021c) is perhaps the closest work to ours among those methods for from-
scratch training. However, conventional MA requires authentic parallel data
among many language pairs, which does not always guarantee to exist. In
comparison, we focus on a more recent fine-tuning setting on popular pre-
training models as well as a realistic setting with no presumption on the
existence of the additional parallel data.

###  4.2 Agreement-based Learning

Agreement-based learning has been proven as a useful paradigm in the language
community (Liang et al., 2006, 2007; Cheng et al., 2016;
NEURIPS2021_5a66b9205). The core idea is to minimize the difference in the
representations between the inputs with the same meaning. Some multilingual
pre-training methods such as Chi et al. (2021) are relevant to agreement-based
learning in the way that they shrink the distance of cross-lingual
representations between parallel data. Zhang et al. (2019) proposed to enforce
an agreement on the output with left-to-right and right-to-left inputs on
recurrent neural networks for machine translation. Yang et al. (2020) proposed
to use phrase-level agreement for machine translation.

Still, Yang et al. (2021c) is the closest work to ours, which encourages
agreement between parallel data in different languages to have the same
translation outputs. Our work creates synthetic data and employs bidirectional
KL loss to enforce the multilingual agreement bidirectionally.

##  5 Conclusions

Despite the fact that multilingual agreement (MA) has shown its effectiveness
in from-scratch training for MNMT, the conventional MA has at least two
shortages that limit its usages: (i) needs authentic extra parallel data,
which can be often unrealistic and (ii) has an ambiguous direction for
agreement-based learning. We propose BMA-SBT as a novel and universal fine-
tuning framework for pre-trained MT models that (i) exempts the need for
authentic parallel data by creating synthetic parallel text written in a
different source language and (ii) specifies the direction of agreement-based
learning with bidirectional KL divergence loss. Experimental results on three
multilingual machine translation datasets illustrate that BMA-SBT can
obviously improve the strong pre-trained baseline system. An in-depth
investigation indicates that BMA-SBT brings additive improvements to the
conventional BT methods for neural machine translation.

Figure 3: Averaged BlonDe scores from six directions in (X $\rightarrow$ En)
on the dataset of TED Talks evaluated with BMA-SBT and the Baseline Model
(Document-level).

## Limitations

The proposed method requires generating synthetic auxiliary parallel data
using translation models, which requires extra computational costs. The
proposed method requires generating synthetic auxiliary parallel data using
translation models, which requires extra computational costs.

#### Large Language Models

Large Language Models (LLMs) such as ChatGPT have shown good translation
abilities (Lu et al., 2023), while they still lag behind supervised systems
(Jiao et al., 2023; Zhu et al., 2023). We do not directly compare them, as
they are much larger in their number of parameters than the systems described
in this work.

## Ethics Statement

We honour and support the EMNLP Code of Ethics. The datasets used in this work
are well-known and widely used, and the dataset pre-processing does not make
use of any external textual resource. In our view, there is no known ethical
issue. End-to-end pre-trained generators are also used, which are subjected to
generating offensive context. But the above-mentioned issues are widely known
to commonly exist for these models. Any content generated does not reflect the
view of the authors.

## References

  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, and Eneko Agirre. 2020\.  A call for more rigor in unsupervised cross-lingual learning.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7375–7388, Online. Association for Computational Linguistics.
  * Bao et al. (2021) Guangsheng Bao, Yue Zhang, Zhiyang Teng, Boxing Chen, and Weihua Luo. 2021.  G-transformer for document-level machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3442–3455, Online. Association for Computational Linguistics.
  * Cettolo et al. (2015) Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. 2015.  The IWSLT 2015 evaluation campaign.  In _Proceedings of the 12th International Workshop on Spoken Language Translation: Evaluation Campaign_ , pages 2–14, Da Nang, Vietnam.
  * Cheng et al. (2016) Yong Cheng, Shiqi Shen, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016\.  Agreement-based joint training for bidirectional attention-based neural machine translation.  In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence_ , IJCAI’16, page 2761–2767. AAAI Press.
  * Chi et al. (2021) Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021.  InfoXLM: An information-theoretic framework for cross-lingual language model pre-training.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3576–3588, Online. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Fan et al. (2022) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2022.  Beyond english-centric multilingual machine translation.  _J. Mach. Learn. Res._ , 22(1).
  * Jiang et al. (2022) Yuchen Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Jian Yang, Haoyang Huang, Rico Sennrich, Ryan Cotterell, Mrinmaya Sachan, and Ming Zhou. 2022.  BlonDe: An automatic evaluation metric for document-level machine translation.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1550–1565, Seattle, United States. Association for Computational Linguistics.
  * Jiao et al. (2023) Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023.  Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine.  _arXiv e-prints_ , page arXiv:2301.08745.
  * Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _International Conference on Learning Representations_.
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A parallel corpus for statistical machine translation.  In _Proceedings of Machine Translation Summit X: Papers_ , pages 79–86, Phuket, Thailand.
  * Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017.  Six challenges for neural machine translation.  In _Proceedings of the First Workshop on Neural Machine Translation_ , pages 28–39, Vancouver. Association for Computational Linguistics.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Lee et al. (2022) Chia-Hsuan Lee, Aditya Siddhant, Viresh Ratnakar, and Melvin Johnson. 2022.  DOCmT5: Document-level pretraining of multilingual language models.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 425–437, Seattle, United States. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020.  Pre-training via paraphrasing.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS’20, Red Hook, NY, USA. Curran Associates Inc.
  * Liang et al. (2007) Percy Liang, Dan Klein, and Michael I. Jordan. 2007.  Agreement-based learning.  In _Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 3-6, 2007_ , pages 913–920. Curran Associates, Inc.
  * Liang et al. (2006) Percy Liang, Ben Taskar, and Dan Klein. 2006.  Alignment by agreement.  In _Proceedings of the Human Language Technology Conference of the NAACL, Main Conference_ , pages 104–111, New York City, USA. Association for Computational Linguistics.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Lu et al. (2022) Hongyuan Lu, Haoyang Huang, Shuming Ma, Dongdong Zhang, Wai Lam, and Furu Wei. 2022.  TRIP: Triangular Document-level Pre-training for Multilingual Language Models.  _arXiv e-prints_ , page arXiv:2212.07752.
  * Lu et al. (2023) Hongyuan Lu, Haoyang Huang, Dongdong Zhang, Haoran Yang, Wai Lam, and Furu Wei. 2023.  Chain-of-Dictionary Prompting Elicits Translation in Large Language Models.  _arXiv e-prints_ , page arXiv:2305.06575.
  * Ma et al. (2021) Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, and Furu Wei. 2021.  DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.  _arXiv e-prints_ , page arXiv:2106.13736.
  * NLLB-Team (2022) NLLB-Team. 2022.  No language left behind: Scaling human-centered machine translation.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Reid and Artetxe (2022) Machel Reid and Mikel Artetxe. 2022.  PARADISE: Exploiting parallel data for multilingual sequence-to-sequence pretraining.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 800–810, Seattle, United States. Association for Computational Linguistics.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Improving neural machine translation models with monolingual data.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 86–96, Berlin, Germany. Association for Computational Linguistics.
  * Sun et al. (2022) Zewei Sun, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Lei Li. 2022.  Rethinking document-level neural machine translation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3537–3548, Dublin, Ireland. Association for Computational Linguistics.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.  Sequence to sequence learning with neural networks.  In _Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2_ , NIPS’14, page 3104–3112, Cambridge, MA, USA. MIT Press.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Yang et al. (2021a) Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan, Xia Song, and Furu Wei. 2021a.  Multilingual machine translation systems from Microsoft for WMT21 shared task.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 446–455, Online. Association for Computational Linguistics.
  * Yang et al. (2021b) Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan, Xia Song, and Furu Wei. 2021b.  Multilingual machine translation systems from Microsoft for WMT21 shared task.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 446–455, Online. Association for Computational Linguistics.
  * Yang et al. (2021c) Jian Yang, Yuwei Yin, Shuming Ma, Haoyang Huang, Dongdong Zhang, Zhoujun Li, and Furu Wei. 2021c.  Multilingual agreement for multilingual neural machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 233–239, Online. Association for Computational Linguistics.
  * Yang et al. (2020) Mingming Yang, Xing Wang, Min Zhang, and Tiejun Zhao. 2020.  Incorporating phrase-level agreement into neural machine translation.  In _Natural Language Processing and Chinese Computing: 9th CCF International Conference, NLPCC 2020, Zhengzhou, China, October 14–18, 2020, Proceedings, Part I_ , page 416–428, Berlin, Heidelberg. Springer-Verlag.
  * Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016\.  Hierarchical attention networks for document classification.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1480–1489, San Diego, California. Association for Computational Linguistics.
  * Zhang et al. (2019) Zhirui Zhang, Shuangzhi Wu, Shujie Liu, Mu Li, Ming Zhou, and Tong Xu. 2019.  Regularizing neural machine translation by target-bidirectional agreement.  In _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence_ , AAAI’19/IAAI’19/EAAI’19. AAAI Press.
  * Zhu et al. (2023) Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023.  Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis.  _arXiv e-prints_ , page arXiv:2304.04675.

##  Appendix A Number of Model Parameters

Model | Number of Parameters
---|---
M2M-100 | 418M
mBART | 611M
MARGE | 963M
mT5 | 1.23B∗
DOCmT5 | 1.23B∗
Baseline Model | 862M
BMA-SBT (Ours) | 862M
Table 5:  Comparison in the number of parameters for the pre-trained models
used in our experiments. $*$: these models all use the model architecture of
mT5-Large, and we report the number of model parameters taken from the
original paper of mT5 reported by Xue et al. (2021).

Table 5 presents the number of model parameters for the pre-trained models
used in our experiments.


# PU-MFA : Point Cloud Up-sampling via Multi-scale Features Attention

Hyungjun Lee
Graduate School of Automotive Engineering
Kookmin University
Seoul
<EMAIL_ADDRESS>
&Sejoon Lim
Department of Automobile and IT Convergence
Kookmin University
Seoul
<EMAIL_ADDRESS>

###### Abstract

Recently, research using point clouds has been increasing with the development
of 3D scanner technology. According to this trend, the demand for high-quality
point clouds is increasing, but there is still a problem with the high cost of
obtaining high-quality point clouds. Therefore, with the recent remarkable
development of deep learning, point cloud up-sampling research, which uses
deep learning to generate high-quality point clouds from low-quality point
clouds, is one of the fields attracting considerable attention. This paper
proposes a new point cloud up-sampling method called Point cloud Up-sampling
via Multi-scale Features Attention (PU-MFA). Inspired by previous studies that
reported good performance using the multi-scale features or attention
mechanisms, PU-MFA merges the two through a U-Net structure. In addition, PU-
MFA adaptively uses multi-scale features to refine the global features
effectively. The performance of PU-MFA was compared with other state-of-the-
art methods through various experiments using the PU-GAN dataset, which is a
synthetic point cloud dataset, and the KITTI dataset, which is the real-
scanned point cloud dataset. In various experimental results, PU-MFA showed
superior performance in quantitative and qualitative evaluation compared to
other state-of-the-art methods, proving the effectiveness of the proposed
method. The attention map of PU-MFA was also visualized to show the effect of
multi-scale features.

_K_ eywords 3D Vision $\cdot$ Deep-learning $\cdot$ Point Cloud $\cdot$
Attention mechanism $\cdot$ Point Cloud Up-sampling

##  1 Introduction

A point cloud is one of the most popular formats for accurately representing
3D geometric information in robotics and autonomous vehicles. Recently, the
number of studies using point clouds has been increasing with the development
of 3D scanners, such as LiDAR [1, 2]. Along with this trend, there is an
increasing demand for high-quality point clouds that are low-noise, uniform,
and dense. However, the high cost of collecting high-quality point clouds
remains problematic. Therefore, point cloud up-sampling, which generates a
low-noise, uniform, and dense point set from noisy, non-uniform, and sparse
point sets, is an interesting study.

Similar to learning-based image super-resolution studies [3, 4], various
learning-based point cloud up-sampling studies [5, 6, 7] show better
performance than traditional point cloud up-sampling studies [8, 9].
Intuitively, the image super-resolution tasks and the point cloud up-sampling
tasks are similar. Unlike the image super-resolution tasks, which process
regular format images, the point cloud up-sampling tasks, which process
irregular formats, require additional consideration. First, the up-sampled
point set should have a uniform distribution and a dense set of points. Next,
the up-sampled point set should represent the details of the target 3D mesh
surface well [10].

A traditional learning-based point cloud up-sampling study usually consists of
a feature extractor and an up-sampler. In addition, most studies use multi-
scale features or attention mechanisms. PU-Net [11], 3PU [12], and PU-GCN [7]
extract multi-scale features from sparse point sets. These studies reported
excellent performance, but the last feature extracted by the feature extractor
has a limitation in that the details of the sparse point set are diluted
features because the output of each layer is used as the input of the next
layer. Dis-PU [13], PU-EVA [6], and PU-Transformer [5] showed successful
performance using the self-attention mechanism to learn long-range
dependencies between points. However, there is a limit to applying the
attention mechanism with limited information because the key, query, and value
of the self-attention mechanism are generated from the same input.

Focusing on these limitations, this paper proposes PU-MFA, a novel method to
fuse multi-scale features and attention mechanisms. PU-MFA solves point cloud
up-sampling through an attention mechanism that uses an adaptive feature for
each layer. The contributions of this research are as follows:

  * •

This paper proposes a point cloud up-sampling method of U-Net structure using
Multi-scale Features (MFs) adaptively to Global Features (GFs).

  * •

Global Context Refining Attention (GCRA), a structure for effectively
combining MFs and attention mechanisms, is proposed. To the best of the
authors’ knowledge, this is the first MultiHead Cross-Attention (MCA)
mechanism proposed in point cloud up-sampling.

  * •

This study demonstrates the effect of MFs by visualizing the attention map of
GCRA in ablation studies.

This method was compared with various state-of-the-art methods using the
Chamfer Distance (CD), Hausdorff Distance (HD), and Point-to-Surface (P2F)
evaluation metrics for the PU-GAN [10] and the KITTI [14] dataset. As a
result, the effectiveness of this method was confirmed by showing better
performance.

##  2 Related Work

###  2.1 Optimization-Based Point Cloud Up-Sampling

Various optimization-based studies have been performed to generate a dense set
of points from a sparse set. Alexa et al. solved up-sampling by inserting new
points into the Voronoi diagram of the local tangential space computed based
on the moving-least-squares error [8]. Lipman et al. explained the up-sampling
using the Locally Optimal Projection (LOP) operator [9]. In this study, the
points were re-sampled by using $L_{1}$ norm. Huang et al. up-sampled a noisy
and non-uniform set of points using an improved LOP that is a weighted LOP
[15]. Later, Huang et al. proposed an advanced method called Edge-Aware Re-
sampling of a set of points (EAR). The EAR first re-samples the edges and then
uses edge-aware up-sampling to resolve the up-sampling [16].

###  2.2 Learning-Based Point Cloud Up-Sampling

With the successful performance of learning-based image super-resolution, many
studies have proposed a learning-based point cloud up-sampling method.

As with image analysis, many studies have used MFs in point cloud up-sampling.
PU-Net [11], the first attempt at deep learning for point cloud up-sampling,
showed good performance by extracting MFs through hierarchical feature
learning and interpolation based on the framework of PointNet++ [17]. 3PU [12]
performed well using MFs via an Intra-Level Dense connection and Inter-Level
Skip connection. PU-GCN [7] uses MFs extracted by Inception DenseGCN. In this
study, Inception DenseGCN could effectively extract MFs with an InceptionNet-
inspired structure [18].

Because of the advantages of learning the long-range dependency of the self-
attention mechanism, it is used in various point cloud up-sampling studies. In
PU-GAN [10], generators are trained using discriminators that apply a self-
attention mechanism. Pugeo-Net [19] showed good performance by using it in
Feature Recalibration. PU-EVA [6] showed successful up-sampling performance
using an EVA Expansion Unit with the mechanism. Dis-PU [13] performed well
using the Local Refinement Unit with self-attention applied to the generated
point set. PU-Transformer [5], which applied the transformer structure for the
first time in point cloud up-sampling, uses Shifted Channel MultiHead Self-
Attention to show the state-of-the-art performance.

##  3 Problem Description

The problem of generating low-noise, uniform, and dense point set
$Q=\left\\{q_{i}\right\\}_{i=1}^{rN}$ was addressed using the Ground Truth
(GT) point set $D=\left\\{d_{i}\right\\}_{i=1}^{rN}$ and sparse point set
$S=\left\\{s_{i}\right\\}_{i=1}^{N}$, where $N$ is the input patch size and
$r$ is the up-sampling ratio. Figure 1 shows the problem description of this
study.

Figure 1: Illustration of an overview of the proposed method.

##  4 Method

This method consists of a Multi-scale Feature Extractor (MFE), Global Context
Refiner (GCR), Coarse Point Generator (CPG), and Self-Attention Block (SAB).
As shown in Figure 2, MFE extracts MFs, an adaptive feature for use in GCR.
GCR uses MFs to refine GFs adaptively and finally produce $Q^{\prime\Delta}$,
where $Q^{\prime\Delta}$ is defined as
$\mathit{Q^{\prime\Delta}}=\left\\{\mathit{q^{\prime\Delta}}_{i}\right\\}_{i=1}^{rN}$.
CPG generates $Q^{\prime}$ from $S$ and SAB extracts GFs from $Q^{\prime}$,
where $Q^{\prime}$ is defined as
$\mathit{Q^{\prime}}=\left\\{\mathit{q^{\prime}}_{i}\right\\}_{i=1}^{rN}$.
Based on the definitions of $Q^{\prime}$ and $\mathit{Q^{\prime\Delta}}$, $Q$
is formulated as equation (1), where $\oplus$ is an element-wise sum.

$\begin{aligned} Q=Q^{\prime}\oplus
Q^{\prime\Delta}\left(\oplus:\textit{element-wise sum}\right)\end{aligned}$

(1)

Figure 2: Illustration of the proposed framework. Here, 3 is the coordinate
dimension, and $H$ is the depth of the layer. In Multi-scale Feature Extractor
(MFE) and Global Context Refiner (GCR), $C$ is the channel, and $K$ is the
expansion ratio. In Coarse Point Generator (CPG) and Self-Attention Block
(SAB), $C\textquoteright$ is the channel and $K^{\prime}$ is the expansion
ratio.

###  4.1 Multi-scale Feature Extractor

Because the GFs extracted from $Q^{\prime}$ via SAB is a feature extracted
from a set of points in which geometric information about the original input
$S$ is diluted, MFE using Point Transformer (PT) [20], an advanced point cloud
analysis technique, extracts MFs from $S$. As shown in Figure 2, the MFE
consists of $H$ PT, and the set of point-wise features extracted from the
$h^{th}$ PT is $F_{h}\in\mathbb{R}^{N\times K^{h-1}C}$. MFs are the set of
$F_{h}$ extracted from all layers of the MFE. The extracted MFs and $F_{h}$
are formulated as in equation (2), where $f_{i}^{h}$ is a point-wise feature
extracted from the $h^{th}$ PT.

$\begin{aligned}
&F_{h}=\left\\{f_{i}^{h}\right\\}_{i=1}^{N},MFs=\left\\{F_{h}\right\\}_{h=1}^{H}\end{aligned}$

(2)

####  4.1.1 Point Transformer

PT consists of two elements. The first is the K-Nearest Neighbor (KNN), and
the second is the Vector Self-Attention (VSA) mechanism. At the $h^{th}$ PT,
the point-wise feature $f_{i}^{h-1}$ of point $s_{i}\in S$ is updated to
$f_{i}^{h}$ through VSA, which uses $s_{i}$ and $patch_{i}$ as the inputs. The
$patch_{i}$ is generated through KNN using $s_{i}$ as the input. This
operation works on all points in $S$, updating the point-wise feature of all
points [20]. This is formulated in equation (3), where $patch\\_size$ is the
size of K in KNN.

$\begin{aligned} &patch_{i}=KNN(s_{i},patch\\_size)\\\
&f_{i}^{h}=\sum_{p_{k}\in patch_{i}}VectorSelfAttention(s_{i},p_{k})\\\
&(s_{i}\in S,i\in\left\\{1,2,...,N\right\\})\end{aligned}$

(3)

Inspired by this operation, this considered $patch_{i}$ is equivalent to the
CNN’s kernel. In CNN, even if the kernel of the CNN is fixed, a deeper layer,
means a wider receptive field. Therefore, even if the patch size of the KNN in
PT is fixed, the deeper the layer, the more $s_{i}$ can interact with a wider
range of points. Figure 3 is an example with a KNN patch size of four. In
Figure 3 (a), when the $h^{th}$ PT updates $f_{i}^{h-1}$ to $f_{i}^{h}$, VSA
is performed on $patch_{i}$, which is composed of $s_{i}$ and incidental
points, to update $f_{i}^{h-1}$. In Figure 3 (b), the $h^{th}$ PT updates each
feature by performing a VSA for each patch in all incidental points. In Figure
3 (c), the $(h+1)^{th}$ PT updates $f_{i}^{h}$ to $f_{i}^{h+1}$ by performing
VSA using $patch_{i}$ similar to the $h^{th}$ PT. However, the $(h+1)^{th}$ PT
updates $f_{i}^{h}$ to $f_{i}^{h+1}$ using a wider receptive field than the
receptive field of the $h^{th}$ PT because the features of the incidental
points of the $(h+1)^{th}$ PT are updated by the $h^{th}$ PT. This operation
allows the MFE to extract the MFs effectively.

(a)

(b)

(c)

Figure 3: Illustration of KNN and VSA in PT.

###  4.2 Global Context Refiner

Because GCR and MFE are U-Net [21] structures, they are composed of $H$ GCRA.
GCRA effectively refines GFs by querying MFs, which is adaptive geometric
information applied to each layer. As shown in Figure 2, the $(H-h+1)^{th}$
GCRA generates $RGC_{H-h+1}\in\mathbb{R}^{N\times K^{h-2}C}$ by using
$F_{h}\in\mathbb{R}^{N\times K^{h-1}C}$ as a query and $RGC_{H-h}$ as a pool.
However, $\mathbb{R}^{N\times r3}$ was used instead to prevent $RGC_{H}$ from
becoming $\mathbb{R}^{N\times\frac{C}{K}}$. After refining the GFs, the linear
layer was used to perform the transformation. PixelShuffle was then used to
generate the $Q^{\prime\Delta}\in\mathbb{R}^{rN\times 3}$.

####  4.2.1 Global Context Refining Attention

Inspired by Skip-Attention [22], which acts as a communicator between the
encoder and decoder, GCRA uses MFs and GFs to apply the MCA mechanism. In
various studies, self-attention mechanisms are used to extract the features of
point sets or to generate up-sampling point sets [5, 10]. However, the self-
attention mechanism is limited because it uses only limited information due to
the structure in which key, query, and value are generated from the same
input. With these limitations in mind, GCRA in the $H$ hierarchy uses
GFs$\in\mathbb{R}^{N\times K^{\prime}C^{\prime}}$ as the pool (key, values)
and MFs as the queries, progressively refining the GFs through MCA. GCRA
consists of MCA [23], Batch Normalization (BN) [24], and Feed Forward. As
shown in Figure 4, the output shape of applying MCA using the query and pool
is $\mathbb{R}^{N\times F_{p}}$. The pool was then refined by adding the pool
and the MCA output. The BN was used for stable training after addition. Feed
Forward transforms the output of the BN and produces a Refined Global Context
(RGC)$\in\mathbb{R}^{N\times F_{o}}$.

Figure 4: Illustration of Global Context Refining Attention (GCRA). $F_{p}$ is
the pool input channel, $F_{q}$ is the query input channel, and $F_{o}$ is the
output channel.

###  4.3 Coarse Point Generator

CPG generates $Q^{\prime}$. In CPG, PT [20] and PixelShuffle [25, 5] generate
$S^{\prime\Delta}$ from $S$, where, $S^{\prime\Delta}$ is defined as
$\mathit{S^{\prime\Delta}}=\left\\{\mathit{s^{\prime\Delta}}_{i}\right\\}_{i=1}^{rN}$.
The structure of CPG consists of four layers, such as the structure of the
3PU’s Feature Extraction Unit [12]. As shown in Figure 2, to make the final
output into 3D coordinates, first, PT was first used to expand the features,
and then gradually reduce them. Subsequently, PixelShuffle generates 3D
coordinates using those features. $Q^{\prime}$ is generated through the
element-wise sum of the generated $S^{\prime\Delta}$ and
$duplicate(S,r)\in\mathbb{R}^{rN\times 3}$. This process is formulated as
equation (4).

$\begin{aligned}
&duplicate(S,r)=\left\\{\overbrace{s_{i},...,s_{i}}^{\textit{r
times}}\right\\}_{i=1}^{N}\\\
&Q^{\prime}=duplicate(S,r)\oplus\mathit{S^{\prime\Delta}}\end{aligned}$

(4)

###  4.4 Self-Attention Block

Inspired by self-attention, which learns long-range dependency [23], we use
MultiHead Self-Attention (MSA) was used to extract the GFs from $Q^{\prime}$.
As shown in Figure 2, the shape of $Q^{\prime}$ was changed from
$\mathbb{R}^{rN\times 3}$ to $\mathbb{R}^{N\times 3r}$, and the coordinates of
$Q^{\prime}$ were used as features of the original point set $S$. The
GFs$\in\mathbb{R}^{N\times K^{\prime}C^{\prime}}$ was then extracted using the
changed shape $Q^{\prime}$ as the input to the MSA.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 5: Visualization result of $\times 4$ up-sampling on PU-GAN dataset.

##  5 Experimental Settings

###  5.1 Datasets

All methods were trained using the most popular PU-GAN [10] dataset in these
experiments and evaluated using the PU-GAN dataset and the KITTI [14] dataset.
The PU-GAN dataset was a synthetic point cloud dataset produced from 147 3D
meshes, and the KITTI dataset was a real-scanned point cloud dataset collected
using real LiDAR.

The training phase used 120 3D meshes from the PU-GAN dataset. All patches
were generated via the Poisson disk sampling after converting the original
mesh to a point cloud, just like the patch-based up-sampling approach. The
sampling resulted in 24,000 input-output pairs.

In the evaluation phase, 27 3D meshes from the PU-GAN dataset were converted
into point clouds to test the synthetic point up-sampling, and the real-
scanned point up-sampling test was performed using the KITTI dataset. The
generated patches should cover all point sets when evaluating the synthetic
point cloud and real-scanned point cloud up-sampling. After merging each up-
sampled patch, the up-sampled point set was reconstructed by farthest point
sampling. More details can be found at study in PU-GAN [10]. This dataset was
downloaded and used from https://github.com/liruihui/PU-GAN.

###  5.2 Loss Function

In most point cloud reconstruction methods, CD is used as the loss function
[26, 22, 27]. However, it was confirmed empirically that the Density-Aware
Charmfer Distance showed good performance, considering the uniformity of the
points set on the CD. Therefore, the total loss was formulated as equation
(5), where $\alpha$ is linearly interpolated from 0.1 to 1 during training and
$\left\|\cdot\right\|_{2}$ is $L_{2}$ norm.

$\begin{aligned}
&\mathit{Loss(Q^{\prime},Q,D)}=\mathit{L_{CD}(Q^{\prime},D)}+\alpha*\mathit{L_{DCD}(Q,D)}\\\
&\mathit{L_{CD}(Q^{\prime},D)}=\frac{1}{\left|Q^{\prime}\right|}\sum_{x\in
Q^{\prime}}\underset{{y\in
D}}{\mathrm{m}in}\left\|x-y\right\|_{2}+\frac{1}{\left|D\right|}\sum_{y\in
D}\underset{x\in Q^{\prime}}{min}\left\|y-x\right\|_{2}\\\
&\mathit{L_{DCD}(Q,D)}=\frac{1}{\left|Q\right|}\sum_{x\in Q}\underset{y\in
D}{min}(1-e^{-\left\|x-y\right\|_{2}})+\frac{1}{\left|D\right|}\sum_{y\in
D}\underset{x\in Q}{min}(1-e^{-\left\|y-x\right\|_{2}})\end{aligned}$

(5)

Method |  $\boldsymbol{\times}$4(2048 $\boldsymbol{\rightarrow}$ 8192) |  $\boldsymbol{\times}$16(512 $\boldsymbol{\rightarrow}$ 8192)
---|---|---
CD$\mathbf{(10^{-3})}$ | HD$\mathbf{(10^{-3})}$ | P2F$\mathbf{(10^{-3})}$ | #Params(M) | CD$\mathbf{(10^{-3})}$ | HD$\mathbf{(10^{-3})}$ | P2F$\mathbf{(10^{-3})}$ | #Params(M)
Dis-PU | 0.2703 | 5.501 | 4.346 | 2.115 | 1.341 | 28.47 | 20.68 | 2.115
PU-EVA | 0.2969 | 4.839 | 5.103 | 2.198 | 0.8662 | 14.54 | 15.54 | 2.198
PU-Transformer | 0.2671 | 3.112 | 4.202 | 2.202 | 1.034 | 21.61 | 17.56 | 2.202
PU-MFA(Ours) | 0.2326 | 1.094 | 2.545 | 2.172 | 0.5010 | 5.414 | 9.111 | 2.172

Table 1: Comparing the quantitative evaluation of $\times 4$ and $\times 16$
up-sampling with the state-of-the-art methods.

###  5.3 Metric

This study evaluated the method using CD, HD, and P2F metrics, as in previous
studies [13, 6, 5]. CD is a metric that measures the similarity between a set
of GT points and a set of predicted points for each point, and HD is an
evaluation metric that measures the outliers in a set of predicted points
based on a set of GT points. P2F is an index that measures the similarity
between the original mesh and the predicted point set and measures the quality
of the predicted point set. The parameter complexity was also measured by
measuring the number of parameters. For all metrics, a lower the number, meant
better performance.

###  5.4 Comparison Methods

The proposed method was compared with three state-of-the-art methods: Dis-pu
[13], PU-EVA [6], and PU-Transformer [5] to validate the method. For an exact
comparison, all methods were implemented using pytorch [28] version 1.7.0 on
Ubuntu 20.04 and trained on the same Intel i9-10980XE CPU and NVIDIA TITAN RTX
environment.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 6: Visualization result of $\times 16$ up-sampling of PU-GAN dataset.

###  5.5 Implementation Details

All methods for the experiment were trained with a batch size of 64 for 100
epochs, and the Adam [29] optimizer with a learning rate of 0.0001 was used.
The patch size of KNN used in PT is set to 20 as in PU-Transformer [5].
Rotation, scaling, random perturbation, and regularization were applied to the
training dataset. as in previous studies [11, 10]. The up-sampling ratio $r$
was four and the input patch size $N$ was 256. The CPG’s $C^{\prime}$ and
$K^{\prime}$ were 32 and 8, respectively. For MFE and GCR, $C$ and $K$ were 16
and 4, respectively. The layer depth of MFE and GCR, $H$, was four. The head
number of MCA and MSA was set to eight, as in the previous study [23].

##  6 Experimental Results

Dis-PU [13], PU-EVA [6], and PU-Transformer [5], and the present method were
compared using the PU-GAN [10] and the KITTI [14] datasets.

###  6.1 Results on 3D Synthetic Datasets

Table 1 lists the quantitative performance comparisons for $\times 4$ and
$\times 16$ up-sampling. $\times 4$ up-sampling sampled 2,048 points to 8,192
points. $\times 16$ up-sampling sampled 512 points to 8,192 points by
repeating the $\times 4$ up-sampling twice. As shown in Table 1, the present
method showed good performance compared to the other state-of-the-art methods.
Although it is not the most efficient in parameter complexity, it showed good
efficiency compared to the performance of the present method.

Figure 5 presents the visualization result of $\times 4$ up-sampling, and
Figure 6 is the visualization result of $\times 16$ up-sampling. Figures 5
(b), (c), and (d), show the set of points representing the bird’s leg, the
space between the kitten’s body and tail, the statue’s leg, and the camel’s
hoof with noisy or unclear boundaries. However, Figure 5 (e) shows low-noise
and clear boundaries. In Figure 6 (b) and (d), the chair back does not
represent the original shape well, and (c) maintains the shape to some extent,
but there is considerable noise. On the other hand, Figure 6 (e) has
relatively little noise and represents the original shape well.

###  6.2 Results on Real-scanned Datasets

Dis-PU, PU-EVA, PU-Transformer, and the present method were evaluated using
the KITTI dataset for $\times 4$ up-sampling. Figure 7 shows $\times 4$ up-
sampling. In Figure 7 (b), (c), and (d), the boundary between the window and
the door of the vehicle was unclear. However, Figure 7 (e) generated by the
present method, showed that the boundary was clearer.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 7: Visualization result of $\times 4$ up-sampling of the KITTI dataset.

###  6.3 Ablation Study

This method, was evaluated by performing various ablation studies using the
PU-GAN dataset.

####  6.3.1 Effect of Components

To demonstrate the effectiveness of the contribution, four cases were divided
into ablation studies. The cases were as follows. : Case 1 was a structure
using GCR, CPG, and SAB, with the MultiHead Attention (MHA) of GCR and SAB
consisting of self-attention with one head. Case 2 was a structure changed
from Case 1 to eight heads. Case 3 was a structure using GCRA composed of MCA
by adding MFE to Case 2, where the query of all GCRA becomes $F_{4}$, the
final output of MFE. Case 4 was PU-MFA. As shown in Table 2, all contributions
affected the method performance.

Case | Contribution | Metric
---|---|---
MHA | MFE | MFs | CD$\mathbf{(10^{-3})}$ | HD$\mathbf{(10^{-3})}$ | P2F$\mathbf{(10^{-3})}$
1 |  |  |  | 0.3349 | 4.461 | 4.926
2 | $\surd$ |  |  | 0.2473 | 1.101 | 2.829
3 | $\surd$ | $\surd$ |  | 0.2500 | 2.735 | 2.737
4 | $\surd$ | $\surd$ | $\surd$ | 0.2362 | 1.094 | 2.545

Table 2: Ablation study results to analyze the effect of the present
contribution.

####  6.3.2 Multi-scale Features Attention Analysis

By visualizing the attention maps of all GCRAs, it was confirmed that the
GCRAs of GCR with $H=4$ refined the GFs by adaptively using the MFs extracted
from receptive fields of various sizes. Figure 8 shows the results visualized
by choosing three attention heads in the GCRA and selecting 30 points, which
had the highest attention score in $S$, from each head. The attention map was
visualized using Case 3 in Table 2 without MFs in Figure 8 (b) to compare that
MFs operated adaptively. As shown in Figure 8 (a), in the low-layer GCRA, an
attention map was formed for a wide range of points in a point set, and in
high-layer GCRA, an attention map was formed for a relatively narrow range of
points. On the other hand, in Figure 8 (b), a wide range of attention maps was
formed regardless of the high and low levels of the hierarchy. This phenomenon
confirmed that PU-MFA uses the adaptive point feature for each layer of the
GCRA.

(a)

(b)

Figure 8: Visualization of attention map generated using MFs as a query in GCR
with $H=4$. Figure 9: Visualization result of the effect of noise.

####  6.3.3 Effect of Noise

Table 3 lists the $\times 4$ up-sampling results of Dis-PU [13], PU-EVA [6],
PU-Transformer [5], and the present method using the PU-GAN dataset with
various noises added. The noise effect evaluated the result obtained by adding
different levels of Gaussian noise $\mathcal{N}$(0,noise level) to a set of
input points. As shown in Table 3, the proposed method showed the most
robustness to various noise levels. As shown in Figure 9, it can be seen that
the boundary between the fingers blurred in the dense set of points generated
by the state-of-the-art methods as the noise level was increased. On the other
hand, the proposed method showed that the boundary between the fingers was
maintained in the dense set of points generated by the present method.

Method | Various noise levels test at x4 Up-sampling(CD with $\mathbf{10^{-3}}$)
---|---
0 | 0.001 | 0.005 | 0.01 | 0.015 | 0.02
Dis-PU | 0.2703 | 0.2751 | 0.2975 | 0.3257 | 0.3466 | 0.3706
PU-EVA | 0.2969 | 0.2991 | 0.3084 | 0.3167 | 0.3203 | 0.3268
PU-Transformer | 0.2671 | 0.2717 | 0.2905 | 0.3134 | 0.3331 | 0.3585
PU-MFA(Ours) | 0.2326 | 0.2376 | 0.2547 | 0.2764 | 0.2989 | 0.3195

Table 3: Quantitative evaluation results of the noise effects using the PU-GAN
dataset.

##  7 Conclusion

In this paper, we proposed PU-MFA, a point cloud up-sampling method of U-Net
structure that combines multi-scale features and attention mechanism. One of
the most significant differences from the previous point cloud up-sampling
methods was that PU-MFA used multi-scale features adaptively and effectively
through fusion with the cross-attention mechanism. Also, the PU-MFA is the
first method to apply the cross-attention mechanism to point cloud up-sampling
to the best of the authors’ knowledge. Various experiments were performed on
PU-MFA and other state-of-the-art methods using the PU-GAN and the KITTI
dataset. As a result, PU-MFA showed better performance than other state-of-
the-art methods in various experiments. In addition, ablation study showed
that multi-scale features are very useful in PU-MFA for generating high-
quality point sets by choosing receptive field size adaptively for each layer.

Despite the successful performance of PU-MFA, PU-MFA cannot cope with an
arbitrary up-sampling ratio. A method that can respond to an arbitrary up-
sampling ratio is planned in the future to overcome this limitation.

##  8 Acknowledgements

This research was supported by BK21 Program(5199990814084) through the
National Research Foundation of Korea (NRF) funded by the Ministry of
Education and Korea Institute of Police Technology (KIPoT) grant funded by the
Korea government (KNPA) (No.092021C26S03000, Development of infrastructure
information integration and management technologies for real time traffic
safety facility operation).

## References

  * [1] Kenji Koide, Jun Miura, and Emanuele Menegatti.  A portable three-dimensional lidar-based system for long-term and wide-area people behavior measurement.  International Journal of Advanced Robotic Systems, 16(2):1729881419841532, 2019.
  * [2] Hyungtae Lim, Suyong Yeon, Soohyun Ryu, Yonghan Lee, Youngji Kim, Jaeseong Yun, Euigon Jung, Donghwan Lee, and Hyun Myung.  A single correspondence is enough: Robust global registration to avoid degeneracy in urban environments.  arXiv preprint arXiv:2203.06612, 2022.
  * [3] Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen.  Single image super-resolution via a holistic attention network.  In European conference on computer vision, pages 191–207. Springer, 2020.
  * [4] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte.  Swinir: Image restoration using swin transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1833–1844, 2021.
  * [5] Shi Qiu, Saeed Anwar, and Nick Barnes.  Pu-transformer: Point cloud upsampling transformer.  arXiv preprint arXiv:2111.12242, 2021.
  * [6] Luqing Luo, Lulu Tang, Wanyi Zhou, Shizheng Wang, and Zhi-Xin Yang.  Pu-eva: An edge-vector based approximation solution for flexible-scale point cloud upsampling.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16208–16217, 2021.
  * [7] Guocheng Qian, Abdulellah Abualshour, Guohao Li, Ali Thabet, and Bernard Ghanem.  Pu-gcn: Point cloud upsampling using graph convolutional networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11683–11692, 2021.
  * [8] Marc Alexa, Johannes Behr, Daniel Cohen-Or, Shachar Fleishman, David Levin, and Claudio T. Silva.  Computing and rendering point set surfaces.  IEEE Transactions on visualization and computer graphics, 9(1):3–15, 2003.
  * [9] Yaron Lipman, Daniel Cohen-Or, David Levin, and Hillel Tal-Ezer.  Parameterization-free projection for geometry reconstruction.  ACM Transactions on Graphics (TOG), 26(3):22–es, 2007.
  * [10] Ruihui Li, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng.  Pu-gan: a point cloud upsampling adversarial network.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7203–7212, 2019.
  * [11] Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng.  Pu-net: Point cloud upsampling network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2790–2799, 2018.
  * [12] Wang Yifan, Shihao Wu, Hui Huang, Daniel Cohen-Or, and Olga Sorkine-Hornung.  Patch-based progressive 3d point set upsampling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5958–5967, 2019.
  * [13] Ruihui Li, Xianzhi Li, Pheng-Ann Heng, and Chi-Wing Fu.  Point cloud upsampling via disentangled refinement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 344–353, 2021.
  * [14] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  * [15] Hui Huang, Dan Li, Hao Zhang, Uri Ascher, and Daniel Cohen-Or.  Consolidation of unorganized point clouds for surface reconstruction.  ACM transactions on graphics (TOG), 28(5):1–7, 2009.
  * [16] Hui Huang, Shihao Wu, Minglun Gong, Daniel Cohen-Or, Uri Ascher, and Hao Zhang.  Edge-aware point set resampling.  ACM transactions on graphics (TOG), 32(1):1–12, 2013.
  * [17] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  Advances in neural information processing systems, 30, 2017.
  * [18] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  * [19] Yue Qian, Junhui Hou, Sam Kwong, and Ying He.  Pugeo-net: A geometry-centric network for 3d point cloud upsampling.  In European Conference on Computer Vision, pages 752–769. Springer, 2020.
  * [20] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun.  Point transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16259–16268, 2021.
  * [21] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [22] Xin Wen, Tianyang Li, Zhizhong Han, and Yu-Shen Liu.  Point cloud completion by skip-attention network with hierarchical folding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1939–1948, 2020.
  * [23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [24] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * [25] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874–1883, 2016.
  * [26] Anh-Duc Nguyen, Seonghwa Choi, Woojae Kim, and Sanghoon Lee.  Graphx-convolution for point cloud deformation in 2d-to-3d conversion.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8628–8637, 2019.
  * [27] Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou.  Pointr: Diverse point cloud completion with geometry-aware transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12498–12507, 2021.
  * [28] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [29] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.